Rewrite lock, copy-on-write

Rewrite lock. Lock a data, then write it, then release the lock. copy-on-write, copy the data into a new place, update the date in new place. Then update the data’s reference to the new place.

Write ahead log(WAL)

WAL persistents operation to disk, then write to cache. If each operation needs to persistent to disk, then it is low efficient. Instead, do batching, this helps to improve performance, also reduce the error to batch level. https://martinfowler.com/articles/patterns-of-distributed-systems/wal.html Flushing every log write to the disk gives a strong durability guarantee (which is the main purpose… Read More »

mysql on mac

1. restart /usr/local/bin/mysql.server restart 2. show mysql variables mysqladmin variables

Two ways to read aws access key/secret. Different CredentialProvider

instance can have instance role. When an application run in instance, it can InstanceProfileCredentialsProvider() to retrieve instance role and have the access. For example, instance_profile from EMR cluster is the role for EMR instance. StsAssumeRoleSessionCredentialsProvider AWSStaticCredentialsProvider AWSCredentialsProviderChain. It will test different credential one by one, until it finds one. Here is a code example

Category: aws

Policies setting: aws account is able to upload to s3

Assume Role Way create a aws user. This user doesn’t has any policy. 2. After created user account, it should tell you the ACCESS_ID and ACCESS_SECRET, copy that to somewhere. 3. create iam role. This role needs a policy, which has access to s3 bucket 4. This role should also have trust relationship with user… Read More »

Category: aws

What is bad code, OO design from Uncle Bob

rigidity code, modules are coupled. Change in module1, requires change module2, then requires change in module3. fragile code, change in module1, but it caused issue in other module or system, which is very unrelated to the module1. Bazzare break, weird break. Like your car window can’t be opened, mechanics fixed the window, but the car… Read More »

Bastion host configuration and private key in ~/.ssh folder,

We need to ssh to bastion host, from there, ssh to xxx.ec2.internal host. The configuration in ~/.ssh/config file is like below: Host *.ec2.internal     // it applies to every *.ec2.internal User hadoop     // the default username for final host. hadoop@xxx.ec2.internal, IdentityFile ~/.ssh/ssh-private.key    // the private ssh key UseKeychain yes ProxyCommand ssh username@xxx.bastion-host.com -W %h:%p. … Read More »

Category: aws

IAM assume policy, IAM policy

1. Create an iam role. During creating the iam role, define IAM assume role policy. Assume policy tells who can assume this iam role. 2. Define IAM Policy. Attach IAM policy to this role. In aws UI, the assume policy will be shown as Trust relationships tab, The normal IAM policy is shown as in… Read More »

Category: aws