|
@jschauma | |||||
|
59. A junior engineer asking "why" and pointing out the docs don't reflect reality is at least as valuable as the senior engineer working blindly off tribal knowledge.
|
||||||
|
||||||
|
Jan Schaumann
@jschauma
|
25. sij |
|
42. "Ancient" is a very relative term when it comes to software and protocols.
43. "Obsolete" doesn't mean it's not in use and relied on.
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
44. The sets of systems online before and after a data center power outage only intersect. Some of the old systems coming online will immediately cause a different outage.
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
45. Some of your most critical services are kept alive by a handful of people whose job description does not mention those services at all.
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
46. After the initial "down for everybody or just me ermahgehrd Slack is down" drop, productivity increases linearly throughout the the duration of the outage.
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
47. You're bound by the CAP theorem much more often than you may think. Halting Problem's a bitch, too.
48. Eventual consistency doesn't help when the system you're debugging hasn't converged yet.
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
49. The source you're looking at is not the code running in production.
50. strace(1)/ktrace(1) doesn't lie.
51. Unless somebody's been playing LD_PRELOAD games.
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
52. Schrödinger's Backup -- "The condition of any backup is unknown until a restore is attempted." -- is overly optimistic.
53. There's an xkcd for the precise situation you find yourself in. (There's also one for at least half of these.)
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
54. At some point in your career you will implement half of kerberos. Poorly.
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
55. Any sufficiently successful product launch is indistinguishable from a DDoS; any sufficiently advanced user indistinguishable from an attacker.
56. Debugging any sufficiently complex open source product is indistinguishable from reverse engineering a black box.
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
57. "We've always done it this way." is not a good reason by itself, but there's bound to be one for why.
58. That reason may or may not be valid any longer, however.
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
60. Your herculean efforts to upgrade the OS across your entire fleet completed just in time for the EOL announcement of the version you upgraded to.
61. This phenomenon was first described in Dante's Inferno as the Ninth Circle of Hell, Ring Four, aka RedHat Canto XXXIV.
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
62. Containers create at least as many problems as they solve.
63. The most ninja move the expert you hired for that third party black box product you rely on is to say "Let me ping the support team".
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
64. Somewhere, somebody ran into this exact problem, but they never bothered to post a solution.
65. That completely automated solution you set up requires at least three manual steps you didn't document.
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
66. CAPEX budget always increases, OPEX budget always decreases.
67. CAPEX costs can be reasonably estimated, OPEX costs can only be ballparked.
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
68. Doubling your time estimate in the hopes of beating expectations won't work because your manager takes your estimate, has a hardy laugh, and then resets it back to what they already promised upchain.
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
69. Your quarterly planning means bubkes when the next re-org rolls around.
70. Most of your actual work is not covered by your OKRs.
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
71. Recursively applying the Pareto Principle is a surprisingly accurate way to gauge your low hanging fruit, determine your high impact objectives, and ballpark your required effort.
72. Although, to be honest, it only works in about 80% of cases.
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
73. Management will always happily spend $$$ on outside consultants to tell them what you've been saying for years.
74. Management will much rather invest in inventing a new, square wheel than fixing an old round one.
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
75. In any organization practicing continuous integration, half of all commits are to fake out CI tests.
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
76. Good software development practices do not always translate well to ops and friends.
77. Mandatory code reviews do not automatically improve code quality nor reduce the frequency of incidents.
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
78. Every new paradigm tends to mostly add layers of abstractions; cutting through them and identifying what basic principles continue to apply is half the battle.
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
79. Real change can only be implemented above layer 7. pic.twitter.com/vQrtYShuP4
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
80. "Prod" is just another name for "staging".
81. Your source of truth lies.
82. Also: it's incomplete.
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
83. pcap or it didn't happen.
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
84. grep(1) > Splunk
(there, I said it)
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
85. Multithreading is rarely worth the added complexity.
86. Parallelism is not Concurrency.
87. Simplicity is King.
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
...and finally...
88. Nobody knows what exactly it is you do.
|
||
|
|
||
|
Jan Schaumann
@jschauma
|
25. sij |
|
This thread as a single HTML page:
netmeister.org/blog/ops-lesso…
Peace out, nerds!
|
||
|
|
||