CategoryUncategorized

Dynamodb Table Export and local Import

If you have problems with serverless framework or another cloud formation setup to create the tables because they already exist there is another option:

  • delete the tables (make a backup)
  • deploy
  • delete the tables (no backup)
  • retore the tables from the backup

Note: The restore could last for hours.

If you really want to do dump and restore continue reading…

Directly into the local Dynamodb

To get your data locally from an aws Dynamodb you should spin your local Dynamodb server

docker run -p 8000:8000 amazon/dynamodb-local

Let’s pull and create the schema in the local.


$TABLE=essential-events
aws dynamodb describe-table \
    --table-name $TABLE | \
    jq '.Table | del(.TableId, .TableArn, .ItemCount, .TableSizeBytes, .CreationDateTime, .TableStatus, .ProvisionedThroughput.NumberOfDecreasesToday)' | \
    tee $TABLE-schema.json

aws dynamodb create-table --cli-input-json file://$TABLE-schema.json --endpoint-url http://localhost:8000

aws dynamodb list-tables --endpoint-url http://localhost:8000

Now as we have the local Dynamodb we can read from the AWS and import into the local.

The tricky part is that the cloud is hostile, you can only import max 25 items and no more than 16mb and so on…

So instead of having a nice export/import one-liners, we have to write a short script.

This script will read 25 records and import them to the local Dynamodb.

TABLE=essential-events
maxItems=25
index=0
DATA=$(aws dynamodb scan --table-name $TABLE --max-items $maxItems)
((index+=1)) 
echo $DATA | jq ".Items | {\"$TABLE\": [{\"PutRequest\": { \"Item\": .[]}}]}" > inserts.jsons
aws dynamodb batch-write-item --request-items file://inserts.jsons --endpoint-url http://localhost:8000


nextToken=$(echo $DATA | jq '.NextToken')
while [[ "${nextToken}" != "" ]]
do
  DATA=$(aws dynamodb scan --table-name $TABLE --max-items $maxItems --starting-token $nextToken)
  ((index+=1))
  echo $DATA | jq ".Items | {\"$TABLE\": [{\"PutRequest\": { \"Item\": .[]}}]}" > inserts.jsons
  aws dynamodb batch-write-item --request-items file://inserts.jsons --endpoint-url http://localhost:8000
  nextToken=$(echo $DATA | jq '.NextToken')
done

By saving it to files

As this probably will be the development table, we would like to do multiple imports from the exported data. Here is a version of the upper script which stores the data in files so that you can re-import later.

TABLE=essential-events
maxItems=25
index=0
DATA=$(aws dynamodb scan --table-name $TABLE --max-items $maxItems)
((index+=1))
echo $DATA | cat > "$TABLE-$index.json"

nextToken=$(echo $DATA | jq '.NextToken')
while [[ "${nextToken}" != "" ]]
do
  DATA=$(aws dynamodb scan --table-name $TABLE --max-items $maxItems --starting-token $nextToken)
  ((index+=1))
  echo $DATA | cat > "$TABLE-$index.json"
  nextToken=$(echo $DATA | jq '.NextToken')
done

Now we have a bunch of table-NNN.json files. When I want to do the import I do

for x in `ls *$TABLE*.json`; do
  cat $x | jq ".Items | {\"$TABLE\": [{\"PutRequest\": { \"Item\": .[]}}]}" > inserts.jsons
  aws dynamodb batch-write-item --request-items file://inserts.jsons
done

How to “truncate” Dynamodb table

$TABLE=essential-events
aws dynamodb describe-table \
    --table-name $TABLE | \
    jq '.Table | del(.TableId, .TableArn, .ItemCount, .TableSizeBytes, .CreationDateTime, .TableStatus, .ProvisionedThroughput.NumberOfDecreasesToday)' | \
    tee $TABLE-schema.json
aws dynamodb delete-table --table-name $TABLE
aws dynamodb create-table --cli-input-json file://$TABLE-schema.json 

Credits for the truncation go to https://medium.com/@samnco/deleting-content-in-dynamodb-from-the-cli-831ce5ab083c

Pull remote files from sftp server

Those days we almost use cloud for everthing. But sometimes we need to pull files from sftp server. Here are two solutions for that

Pull and remove with sftp

This solution pulls the files then removes them from the remote. There is a gotcha that if you expect a lot of files there might be a chance a file to arrive while the “get -r …” command is executing. Then the “rm *” will remove it. So this is suitable if you expect a few files a week/day

Create a batchfile.sh

get -r upload/* incoming/
rm upload/*

Then add cron

0 5 * * * /usr/bin/sftp -b batchfile.sh [email protected]

Only pulling with lftp

When I don’t have permissions to remove the files from the remote sftp I use the following off-the-shelf aproach.

This cron is synchronizing files all files to /home/USERNAME/incoming

0 5 * * *  /usr/bin/lftp -u USERNAME,none -e 'mirror --newer-than="now-7days" --only-newer --exclude .ssh --only-missing / /home/USERNAME/incoming; quit' sftp://sftp-corp.company.com

deploy pg gem with postgres 10

When in your distribution the postgres is stick to version 10 and you have to upgrade to postgres-11 a good way to do a capistrano deploy is like this

Do the system install with

yum install postgresql10-contrib postgresql10-devel

And then in your /shared/.bundle/config add a line showing the location of the pg libraries

---
BUNDLE_PATH: "/opt/application/shared/bundle"
BUNDLE_BUILD__PG: "--with-pg-config=/usr/pgsql-10/bin/pg_config"
BUNDLE_FROZEN: "true"
BUNDLE_JOBS: "4"
BUNDLE_WITHOUT: "development:test"

Thanks to my colleague Kris for finding the solution.

Organizing terraform modules in application stacks for free

One of the big challenges is how to organize your cloud account setup.

In one account, you can have a couple of application stacks. The challenge is to able fast to apply/plan/destroy them without any burden.

What works for me is to use one application-terraform-bulk.sh script which will know which modules to which stacks belong. And if I have a couple of modules to apply in the application stack I use terraform-bulk.sh script which just applies all modules in the current folder.

Here is an example.

Those are the ECR modules which must be presented in this account. I do not care about which stack will own them, so I will use the general terraform_bulk.sh script

The commands which I can do are:

./terraform_bulk.sh init
# ...It will go in each folder and do terraform init
./terraform_bulk.sh plan
# ...It will go in each folder and do terraform plan
./terraform_bulk.sh apply
./terraform_bulk.sh destroy

Here is how it looks the script

#!/bin/bash
trap "exit" INT

modules=(
  anaconda
  essential
  essential-anaconda-environment
)


terraform_plan() {
  local project="$1"
  pushd .
  cd $project
  terraform plan
  popd
}



terraform_init() {
  local project="$1"
  pushd .
  cd $project
  terraform init
  popd
}


terraform_apply() {
  local project="$1"
  pushd .
  cd $project
  terraform apply -auto-approve
  popd
}


terraform_destroy() {
  local project="$1"
  pushd .
  cd $project
  terraform destroy -auto-approve
  popd
}


terraform_show() {
  local project="$1"
  pushd .
  cd $project
  terraform show
  popd
}


# array=(1 2 3 4)
# reverse array foo
# echo "${foo[@]}"
reverse() {
    # first argument is the array to reverse
    # second is the output array
    declare -n arr="$1" rev="$2"
    for project in "${arr[@]}"
    do
        rev=("$project" "${rev[@]}")
    done
}





case "$1" in
  init)
      for project in "${modules[@]}"
      do
        echo ""
        echo ""
        echo $project
        terraform_init $project
      done

      ;;

  show)
      for project in "${modules[@]}"
      do
        echo ""
        echo ""
        echo $project
        terraform_show $project
      done
      ;;


  apply)
      for project in "${modules[@]}"
      do
        echo ""
        echo ""
        echo $project
        terraform_apply $project
      done
      ;;

  destroy)
      reverse modules reversed_modules
      for project in "${reversed_modules[@]}"
      do
        echo ""
        echo ""
        echo $project
        terraform_destroy $project
      done
      ;;

  plan)
      reverse modules reversed_modules
      for project in "${reversed_modules[@]}"
      do
        echo ""
        echo ""
        echo $project
        terraform_plan $project
      done
      ;;

  *)
      echo $"Usage: $0 {init|apply|destroy}"
      exit 1

esac


In my case in the development cloud account, I have to host two applications. Then I just create two versions of the script like this.

wxr-xr-x 13 guda guda 4096 Nov 15 13:20 .
drwxr-xr-x  6 guda guda 4096 Nov  5 11:20 ..
drwxr-xr-x  3 guda guda 4096 Oct 28 18:14 athena
drwxr-xr-x  3 guda guda 4096 Jul 10 15:17 cm
drwxr-xr-x  5 guda guda 4096 Dec  5 22:42 ecr
drwxr-xr-x 11 guda guda 4096 Oct 28 18:39 endpoints
-rwxr-xr-x  1 guda guda 2345 Oct 28 18:38 essential-terraform_bulk.sh <<<<<
-rwxr-xr-x  1 guda guda 2190 Oct 28 18:14 etl_monitoring-terraform_bulk.sh <<<<<
drwxr-xr-x  3 guda guda 4096 Nov  5 11:24 fargate_essential
drwxr-xr-x  3 guda guda 4096 Oct 28 18:47 rds
drwxrwxr-x  3 guda guda 4096 Sep  3 19:48 s3
drwxr-xr-x  5 guda guda 4096 Oct 28 18:47 secret_manager
drwxr-xr-x  3 guda guda 4096 Aug 15 17:02 vpc
drwxr-xr-x  4 guda guda 4096 Nov 15 13:20 vpc_peering
drwxr-xr-x  3 guda guda 4096 Aug 19 14:51 zone_security_groups

So when I want to provision:

  • essential app – use essential-terraform_bulk.sh
  • etl monitoring app – use etl_monitoring-terraform_bulk.sh

Be aware when you have to share the resources – for example vpc, you do not want the first terraform-bulk.sh to drop a resource which is needed by the second application terraform bulk.

Saboteur – Rules (in Bulgarian)

Saboteur by Frederic Moyersoen

Link to the English version:

Continue reading

How to clean wordpress website

This works for infection with *.buyittraffic.com and *.gotosecond2.com

If you have an ancient WordPress 4.1.1 you and your website has become a victim of cross-site scripting here is how to clean it.

First, update your WordPress to one which is not vulnerable, such version is 4.1.28 which can be downloaded from here

In my case the victim was http://www.YOURWEBSITE.com/ and links were changed to go to http://land.buyittraffic.com

Fix the Links

Go to MySQL cli or your phpMyAdmin and recover the website URL and your home page url:

UPDATE wp_options SET option_value = 'http://www.YOURWEBSITE.com/' WHERE `wp_options`.`option_name` = 'siteurl';

UPDATE wp_options SET option_value = 'http://www.YOURWEBSITE.com/' WHERE `wp_options`.`option_name` = 'home';

This will fix the links on the website and administration will start to work.

At this point, you can open your website but DO NOT CLICK on any link. All posts/pages are infected.

Fix the content

To all of them have been appended nasty javascript and you have to clean them. To find the malware code which we have to delete run a curl command to see the HTML of the webpage. Copy one page/post url and check the source with “curl”

curl https://www.YOURWEBSITE.com/page?id=123

You will see something like this at the end

<script src='https://js.greenlabelfrancisco.com/clizkes' type='text/javascript'></script>
<script src='https://dl.gotosecond2.com/clizkes' type='text/javascript'></script>

Then run those commands in the mysql console:


UPDATE wp_posts 
  SET post_content = REPLACE(post_content,
      "<script src='https://scripts.trasnaltemyrecords.com/pixel.js?track=r&subid=043' type='text/javascript'></script><script src='https://scripts.trasnaltemyrecords.com/pixel.js?track=r&subid=043' type='text/javascript'></script><script src='https://land.buyittraffic.com/clizkes' type='text/javascript'></script>",
      '');


Check to see if you have nasty code with

curl http://www.YOURWEBSITE.com/

If you don’t see the malware then it is safe to open in the browser again.

Check for Adminer

Check to see if your site contain remote administration php

grep -lri Adminer wordpress/

In my case the file was named ad.php

Delete it!

Happy 2020!

Open a pull request on github from the current branch

When you push a new branch to GitHub from the command line, you’ll notice a URL within the output which you can copy in order to quickly open a new pull request.

But if you are on old branch…then nothing will help you.

And I was tired opening the github website so…. here it is a small ruby script which opens my browser at the correct place in github.

#!/usr/bin/env ruby
output = `git branch`
selected_branch = output.split("\n").find{|element| element =~ /^\*/}
current_branch = selected_branch.split(' ').last

remote = `git remote -v`.split("\n").find{|element| element =~ /origin\s.*?push\)$/}.split(' ')[1]

githost = remote.gsub('[email protected]', '').gsub(/.git$/, '').gsub(/^github.com:/, 'github.com/')


`chromium-browser https://#{githost}/compare/#{current_branch}?expand=1`

Update: This version supports sub-modules

 

Open current github repository in Browser

Script to open the current github repo in the browser

#!/usr/bin/env ruby
output = `git branch`
selected_branch = output.split("\n").find{|element| element =~ /^\*/}
current_branch = selected_branch.split(' ').last

remote = `git remote -v`.split("\n").find{|element| element =~ /origin\s.*?push\)$/}.split(' ')[1]

githost = remote.gsub('[email protected]', '').gsub(/.git$/, '').gsub(/^github.com:/, 'github.com/')


`chromium-browser https://#{githost}/`

Track the current branch

The cure is the next script

#!/usr/bin/env ruby
output = `git branch`
selected_branch = output.split("\n").find{|element| element =~ /^\*/}
current_branch = selected_branch.split(' ').last

origin = `git remote -v`.split("\n").find{|element| element =~ /origin\s.*?push\)$/}.split(' ')[0]

`git branch --set-upstream-to=#{origin}/#{current_branch} #{current_branch}`

Great notes on development

https://blog.juliobiason.net/thoughts/things-i-learnt-the-hard-way/

HTTPS Connections counting

Here is how one can setup a nginx to count the https connections made.

Preparation

Create a new folder

mkdir ~/docker_ssl_proxy
cd ~/docker_ssl_proxy

Put a dummy entry in your /etc/hosts file

127.0.0.1 YOURDOMAIN.com

Steps

First generate certificate

openssl req -subj '/CN=YOURDOMAIN.com' -x509 -newkey rsa:4096 -nodes -keyout key.pem -out cert.pem -days 365

create a new file something.conf with the following content

server {
  listen 4000 ssl;
  ssl_certificate /etc/nginx/conf.d/cert.pem;
  ssl_certificate_key /etc/nginx/conf.d/key.pem;

  # access_log /dev/stdout;
  access_log  /var/log/nginx/access.log;
  error_log /var/log/nginx/error.log;

  location / {
      return 200 'With style!';
      add_header Content-Type text/plain;
  }


}

Then run the docker with

docker run --rm -v `pwd`/logs:/var/log/nginx -v `pwd`:/etc/nginx/conf.d -p 4000:4000 nginx

Get the cacert

echo quit | openssl s_client -showcerts -servername server -connect YOURDOMAIN.com:4000 > cacert.pem
curl --cacert cacert.pem https://YOURDOMAIN.com:4000/ -d 'hello world'

And finally do some connections

go-wrk  -c=400 -t=8 -n=10000 -m="POST" -b='{"accountID":"1"}'  -i https://YOURDOMAIN.com:4000

 

Abbreviations lower our performance

I don’t know why people started doing abbreviations, maybe in the past, the bytes were expensive. I suppose life was harsh and there was no enough food for all and the way they named their programs and variables is mirroring their life. Nowadays we have enough goods and time and free space everywhere and we still name our variables/etc. like we are at the dark ages.

My point is that when we are solving some problem is good to have all neurons of our brain to work solving the problem. If we have to decrypt variables, our project structure is not good, we haven’t used with our editor then we are putting bariers which block us of seeing the best solution because our brain is dedicating 5-10% of its power for nonsense.

I am not saying that we should use full sentences of naming the variables/methods/classes/packages/programs. Only that we do not need to spent time decrypting the abbreviation.

I would love to see an operating system where there is no hackish syndrome.

In the cloud

AWS give examples in their documentation with hackish. How it is possible AWS to have so high expectations for hiring developers and let them act as a woodcutter.

The load balancer names in AWS have a size limit in their names so you that you can have YOUR-APP-us-east-1-production load balancer. You have to name it Your-APP-us-east-1-prod.

At Home

My son is learning his computer language and yesterday he asked me what do the method Intn(n Int) – I can’t answer.

“Mom brg me sndwch!”

At Work

Here are some very popular examples

  • dev > development
  • prod > production
  • ctx, ctx > context vs
  • obj > object

Linux

Do you know why we write “mount” to mount some file system, and “umount” to unmount? Why?

The opposite command “mount” is not abbreviated to “mnt” or even “mt”. This inconsistency is crazy!

For RobotsFor Humans
lsblkblock-devices
mountunmount

© 2020 Gudasoft

Theme by Anders NorénUp ↑