Page 3 of 28

Extract a huge number of files from AWS s3 glacier

…or how to run multiple commands in parallel

Continue reading

Dynamodb Table Export and local Import

Continue reading

Pull remote files from sftp server

Those days we almost use cloud for everthing. But sometimes we need to pull files from sftp server. Here are two solutions for that

Pull and remove with sftp

This solution pulls the files then removes them from the remote. There is a gotcha that if you expect a lot of files there might be a chance a file to arrive while the “get -r …” command is executing. Then the “rm *” will remove it. So this is suitable if you expect a few files a week/day

Create a batchfile.sh

get -r upload/* incoming/
rm upload/*

Then add cron

0 5 * * * /usr/bin/sftp -b batchfile.sh [email protected]

Only pulling with lftp

When I don’t have permissions to remove the files from the remote sftp I use the following off-the-shelf aproach.

This cron is synchronizing files all files to /home/USERNAME/incoming

0 5 * * *  /usr/bin/lftp -u USERNAME,none -e 'mirror --newer-than="now-7days" --only-newer --exclude .ssh --only-missing / /home/USERNAME/incoming; quit' sftp://sftp-corp.company.com

deploy pg gem with postgres 10

When in your distribution the postgres is stick to version 10 and you have to upgrade to postgres-11 a good way to do a capistrano deploy is like this

Do the system install with

yum install postgresql10-contrib postgresql10-devel

And then in your /shared/.bundle/config add a line showing the location of the pg libraries

---
BUNDLE_PATH: "/opt/application/shared/bundle"
BUNDLE_BUILD__PG: "--with-pg-config=/usr/pgsql-10/bin/pg_config"
BUNDLE_FROZEN: "true"
BUNDLE_JOBS: "4"
BUNDLE_WITHOUT: "development:test"

Thanks to my colleague Kris for finding the solution.

Organizing terraform modules in application stacks for free

Continue reading

Switch configuration lines using comments

Continue reading

Saboteur – Rules (in Bulgarian)

Saboteur by Frederic Moyersoen

Link to the English version:

Continue reading

How to clean wordpress website

Continue reading

Open a pull request on github from the current branch

Continue reading

Great notes on development

https://blog.juliobiason.net/thoughts/things-i-learnt-the-hard-way/

© 2021 Gudasoft

Theme by Anders NorénUp ↑