INT 21h

Hi, I am Vladimir Smagin, SysAdmin, DevOps and barely good guy. Telegram Email / GIT / Микроблог / Thingiverse / GPG

How to move snap application between computers with snapshots

№ 10960. В разделе " Sysadmin " от March 29th, 2020

В подшивках: ,

wal-g and Postgresql

№ 10938 В разделе "Sysadmin" от March 20th, 2020,
В подшивках: , ,

Initials

Master database: postgresql-10, in production
Backup storage: minio with empty pg-walg-backup bucket

Task: Create slave postgresql-10 server with recovery

Preparing for both servers

Install wal-g binary manually or use my Ubuntu repository (install walg-lzo).

Create special script for wal-g with variables, save to /var/lib/postgresql/walg-postgresql

#!/bin/bash
source /etc/profile

export AWS_ENDPOINT=http://minio.myserv.com:9000
export AWS_ACCESS_KEY_ID="KEYNAMEKEYNAMEKEYNAME"
export AWS_SECRET_ACCESS_KEY="SECRETSECRETSECRETSECRETSECRETSECRET"
export WALE_S3_PREFIX="s3://pg-walg-backup/"
export AWS_S3_FORCE_PATH_STYLE="true"
export AWS_REGION=us-east-1
export PGHOST=/var/run/postgresql
export PGUSER=postgres
export PGPORT=5433

wal-g $@

It is not secure to store this file global readable. Best choice to place it in postgres home directory without read by others:

chmod 500 /var/lib/postgresql/walg-postgresql
chown postgres:postgres /var/lib/postgresql/walg-postgresql

Master server

Add to postgresql.conf:

archive_mode = on
archive_command = '/var/lib/postgresql/walg-postgresql wal-push %p'

Ok, now run backup-push to create full backup:

/var/lib/postgresql/walg-postgresql backup-push /pg_data/10/main

Slave server

Stop postgresql and remove data directory, wal-g requires empty directory before restore.

Now run:

/var/lib/postgresql/walg-postgresql backup-fetch /pgdata/10/main LATEST

Create /pgdata/10/main/recovery.conf:

standby_mode = 'on'
restore_command = 'walg-postgresql wal-fetch "%f" "%p"'

Now see logfile for wal restoring, database in recovery mode.

Нет комментариев »

Postgresql logical backup failed, pg_dump said: PQgetResult() failed

№ 10947 В разделе "Sysadmin" от March 20th, 2020,
В подшивках: , ,

You made streaming backups and restoring with walg and now trying to make logical backups.

But some shit ruined your plans:

pg_dump: Dumping the contents of table "<table>" failed: PQgetResult() failed.

Many sites suggests to enable hot_standby_feedback in postgresql.conf, but actually you need also increase max_standby_archive_delay and max_standby_streaming_delay. I set 1 hour.

Нет комментариев »

Midnight Commander (mc) golden color scheme

№ 10915 В разделе "Sysadmin" от February 16th, 2020,
В подшивках:

My own color scheme for Midnight Commander. Nicely looks only on dark backgrounds and much better with transparent backgrounds.

Installation:

  • Add line export TERM=xterm-256color to the end of ~/.bashrc in your home directory
  • Place slut.ini to home directory ~/.local/share/mc/skins/slut.ini or /usr/share/mc/skins to be system-wide

Download slut.ini

Нет комментариев »

docker-compose for Elasticsearch, Kibana and oauth2 protection

№ 10908 В разделе "Sysadmin" от January 17th, 2020,
В подшивках: ,

version: '3.7'
services:
  kibana:
    image: kibana:7.3.0
    depends_on:
      - elasticsearch
    networks:
      - elk

  elasticsearch:
    image: elasticsearch:7.3.0
    volumes:
      - esdata:/usr/share/elasticsearch/data
    networks:
      - elk
    ports:
      - 39200:9200
    environment:
      - "discovery.type=single-node"
      - "cluster.name=docker-cluster"
      - "bootstrap.memory_lock=true"
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1

  oauth:
    # cloned git repo with enabled bitbucket support
    build: ./oauth2_proxy
    image: oauth2proxy
    entrypoint:
      - oauth2_proxy
      - --upstream=http://kibana:5601
      - --email-domain=*
      - --http-address=0.0.0.0:4180
      - --bitbucket-team=my_organization
      - --client-id=zZYjbsBVMBDyaXvk5v
      - --client-secret=wxz3uFvKVBXR2EaQPJAcQyPY44XbyNKT
      - --provider=bitbucket
      - --cookie-secret=cy-BbEK5MgHg5NcQe8FcdQ==
      - --cookie-secure=true
    depends_on:
      - elasticsearch
      - kibana
    ports:
      - 127.0.0.1:4180:4180
    networks:
      - elk

networks:
  elk:

volumes:
  esdata:
    driver: local

Нет комментариев »

Микроблог перейти

# 2019-12-23 09:14:01

Очень интересная концепция, docker-compose для kubernetes kompose.io/ #devops #kubernetes #docker

# 2019-12-16 17:21:18

Надо будет почитать для общего развития как оживить кластер в случае проеба сертов habr.com/ru/company/southbridg #kubernetes #devops

# 2019-12-12 17:33:46

js bootstrap с интерфейсом как в DOS github.com/kristopolous/BOOTST


© Vladimir Smagin, 2005-2019. Копирование материалов без разрешения запрещено. GPG DA4CD0F5E222EA727D6A40C413BCE12E5618F071 *
Яндекс.Метрика

Fortune cookie: "No man with any sense of humor ever founded a religion." [Robert G. Ingersoll, "What Must We Do To Be Saved", 1880]