Sergey Shinder - DEV Community

Sergey Shinder

Sep 15

Every new replica took twenty more database connections

#sergeyshinder #sre #postgres #reliability

2 min read

Sergey Shinder

Sep 15

A third of our releases went out through the emergency path

#sergeyshinder #releaseengineering #devops #cicd

1

2 min read

Sergey Shinder

Sep 15

Their sandbox accepted everything their production refused

#sergeyshinder #api #integration #testing

2 min read

Sergey Shinder

Sep 15

Nobody could tell me when the latency had started

#sergeyshinder #observability #monitoring #performance

2 min read

Sergey Shinder

Sep 15

A temporary file on one pod evicted every other pod on the node

#sergeyshinder #kubernetes #containers #storage

2 min read

Sergey Shinder

Sep 15

The apply was green and nothing in the environment could get out

#sergeyshinder #terraform #iac #networking

2 min read

Sergey Shinder

Sep 15

The secret was masked until we base64 encoded it

#sergeyshinder #cicd #security #githubactions

2 min read

Sergey Shinder

Sep 13

One instance was bad at its job and nothing was designed to notice

#sergeyshinder #sre #reliability #loadbalancing

2 min read

Sergey Shinder

Sep 13

Our own retention policy deleted the version we needed to roll back to

#sergeyshinder #releaseengineering #docker #devops

1

2 min read

Sergey Shinder

Sep 13

We read the partner's file while they were still writing it

#sergeyshinder #integration #api #sftp

2 min read

Sergey Shinder

Sep 13

Our dashboards were green because the request never reached us

#sergeyshinder #monitoring #observability #frontend

2 min read

Sergey Shinder

Sep 13

The sidecar left before the application had finished

#sergeyshinder #kubernetes #istio #containers

1

2 min read

Sergey Shinder

Sep 13

An environment was two settings that had to agree

#sergeyshinder #terraform #iac #devops

1

2 min read

Sergey Shinder

Sep 13

Each pull request was tested against a main that no longer existed

#sergeyshinder #cicd #git #testing

1

2 min read

Sergey Shinder

Sep 12

Nothing could start without the dependency we had filed as optional

#sergeyshinder #sre #reliability #architecture

1

2 min read

Sergey Shinder

Sep 12

The version number that promised nothing had changed

#sergeyshinder #versioning #semver #automation

1

2 min read

Sergey Shinder

Sep 12

A new status value from our partner rejected the whole batch

#sergeyshinder #api #integration #json

1

2 min read

Sergey Shinder

Sep 12

Somebody silenced that alert in March and it stayed silenced

#sergeyshinder #monitoring #alerting #oncall

2 min read

Sergey Shinder

Sep 12

Our workers were idle and the queue was an hour deep

#sergeyshinder #kubernetes #autoscaling #performance

1

2 min read

Sergey Shinder

Sep 12

We told Terraform to ignore one field and it hid a firewall change

#sergeyshinder #terraform #iac #security

2 min read

Sergey Shinder

Sep 12

Two deploys raced and the older commit won

#sergeyshinder #cicd #githubactions #deployment

1

2 min read

Sergey Shinder

Sep 11

The cold start we had never practised

#sergeyshinder #sre #reliability #incidentresponse

2 min read

Sergey Shinder

Sep 11

The deploy that finished while a third of the fleet stayed behind

#sergeyshinder #deployment #automation #aws

1

2 min read

Sergey Shinder

Sep 11

The 200 OK that meant the payment had failed

#sergeyshinder #api #integration #payments

2

2 min read

Sergey Shinder

Sep 11

The alert that never saw a spike shorter than its own window

#sergeyshinder #prometheus #monitoring #alerting

2

2 min read

Sergey Shinder

Sep 11

The three replicas that turned out to be on one node

#sergeyshinder #kubernetes #scheduling #resilience

1

2 min read

Sergey Shinder

Sep 11

The apply we interrupted and the resources state never heard about

#sergeyshinder #terraform #iac #automation

1

2 min read

Sergey Shinder

Sep 11

The shell step that could not fail no matter what we did to it

#sergeyshinder #cicd #bash #testing

2 min read

Sergey Shinder

Sep 10

The five minutes when both data centres thought they were primary

#sergeyshinder #sre #reliability #database

1

2 min read

Sergey Shinder

Sep 10

The commit we tested was not the commit we shipped

#sergeyshinder #cicd #releaseengineering #automation

2 min read

Sergey Shinder

Sep 10

The API quota we were sharing with a batch job we did not own

#sergeyshinder #api #integration #ratelimiting

2 min read

Sergey Shinder

Sep 10

The log lines we lost exactly when the incident got big

#sergeyshinder #observability #logging #monitoring

1

2 min read

Sergey Shinder

Sep 10

The DNS setting that turned one external call into five lookups

#sergeyshinder #kubernetes #dns #performance

1

2 min read

Sergey Shinder

Sep 10

The provider upgrade that changed a default we never set

#sergeyshinder #terraform #iac #aws

2 min read

Sergey Shinder

Sep 10

The path filter that skipped the tests for the code we changed

#sergeyshinder #cicd #monorepo #testing

2 min read

Sergey Shinder

Sep 9

The ten minute outage that took six hours to finish

#sergeyshinder #sre #reliability #architecture

1

2 min read

Sergey Shinder

Sep 9

The canary that never saw the traffic that would have broken it

#sergeyshinder #releaseengineering #deployment #devops

2 min read

Sergey Shinder

Sep 9

The order ID that changed by one digit between two services

#sergeyshinder #api #integration #json

2 min read

Sergey Shinder

Sep 9

The graph that stayed flat because nothing was reporting any more

#sergeyshinder #observability #monitoring #prometheus

2 min read

Sergey Shinder

Sep 9

The CPU limit that throttled a service using a third of its quota

#sergeyshinder #kubernetes #containers #performance

1

2 min read

Sergey Shinder

Sep 9

The hotfix somebody made by hand that our nightly apply reverted

#sergeyshinder #terraform #iac #automation

2 min read

Sergey Shinder

Sep 9

The required status check that never ran and blocked nothing

#sergeyshinder #cicd #devops #testing

2 min read

Sergey Shinder

Sep 8

The HTTP client with no timeout that took down four services

#sergeyshinder #sre #reliability #incident

1

2 min read

Sergey Shinder

Sep 8

The migration that shipped with the code and locked the table for eleven minutes

#sergeyshinder #releaseengineering #database #deployment

2 min read

Sergey Shinder

Sep 8

The paginated export that silently skipped records every night

#sergeyshinder #api #integration #database

2 min read

Sergey Shinder

Sep 8

The one percent trace sampling that never captured a single failure

#sergeyshinder #observability #tracing #opentelemetry

2 min read

Sergey Shinder

Sep 8

The liveness probe that restarted every pod during a database slowdown

#sergeyshinder #kubernetes #containers #reliability

2 min read

Sergey Shinder

Sep 8

The list element we removed that recreated nine other resources

#sergeyshinder #terraform #iac #infrastructure

2 min read

Sergey Shinder

Sep 8

The CI cache key that let one branch poison every build

#sergeyshinder #cicd #githubactions #devops

1

2 min read

Sergey Shinder

Sep 7

The load test that passed because it was too polite

#sergeyshinder #sre #performance #reliability

1

2 min read

Sergey Shinder

Sep 7

The artifact we built twice and got two different binaries

#sergeyshinder #releaseengineering #buildsystem #automation

1

2 min read

Sergey Shinder

Sep 7

The certificate our partner rotated without telling anyone

#sergeyshinder #api #integration #tls

2 min read

Sergey Shinder

Sep 7

The debug logging we left on in production for a month

#sergeyshinder #logging #observability #monitoring

2 min read

Sergey Shinder

Sep 7

The container that ignored SIGTERM and dropped every in flight request

#sergeyshinder #kubernetes #containers #docker

1

2 min read

Sergey Shinder

Sep 7

The Terraform state file that stored our database password in plain text

#sergeyshinder #terraform #iac #security

2 min read

Sergey Shinder

Sep 7

The pipeline that could deploy to production from any branch

#sergeyshinder #cicd #security #devops

2 min read

Sergey Shinder

Sep 5

The three nines we promised without measuring anything

#sergeyshinder #sre #reliability #monitoring

2 min read

Sergey Shinder

Sep 5

The deploy that needed one person who was on holiday

#sergeyshinder #automation #deployment #devops

2 min read

Sergey Shinder

Sep 5

The optional field every consumer treated as required

#sergeyshinder #api #integration #backend

2 min read

Sergey Shinder

Sep 5

The metric label that cost more than the service

#sergeyshinder #observability #monitoring #prometheus

2 min read

Writing Debut