# DevOps 4: Observability va server asoslari — crash, MetricKit, SLO, VPS

> **Holat:** 2026-yil sentabr. Ilova App Store'ga chiqqandan keyin ish tugamaydi: u foydalanuvchida qanday ishlayotganini ko'rish, muammoni tez topish va backend'ni barqaror ushlash kerak. Bu dars shu haqida.

## Mundarija

1. [Observability nima?](#observability)
2. [iOS'da strukturali log: `os.Logger`](#logger)
3. [Crash reporting va dSYM](#crash)
4. [MetricKit — Apple'ning bepul telemetriyasi](#metrickit)
5. [Xcode Organizer metrikalari](#organizer)
6. [SLI, SLO va error budget](#slo)
7. [Incident va blameless postmortem](#incident)
8. [Kichik backend uchun VPS: real konfiguratsiya](#vps)
9. [Secrets va CI](#secrets)
10. [Xulosa](#xulosa)
11. [Manbalar](#manbalar)

---

## 1. Observability nima? {#observability}

Observability — tizimning ichki holatini uning tashqi signallari orqali tushunish qobiliyati. Uch asosiy signal bor:

| Signal      | Nima                                          | Misol                                                |
| ----------- | --------------------------------------------- | ---------------------------------------------------- |
| **Logs**    | Vaqt belgisi bilan yozilgan alohida hodisalar | `purchase_failed: product=pro_monthly error=network` |
| **Metrics** | Vaqt bo'yicha yig'ilgan raqamlar              | ilova ishga tushish vaqti p90 = 420 ms               |
| **Traces**  | Bitta so'rovning tizim bo'ylab yo'li          | iOS → API → DB → API → iOS, har bosqich necha ms     |

Backend'da bu uchalasini bitta standart — **OpenTelemetry** — orqali yig'ish odat bo'lgan. iOS'da esa asosiy manbalar boshqacha: tizim logi (`os.Logger`), crash reporter (Crashlytics yoki Sentry), **MetricKit** va Xcode Organizer.

---

## 2. iOS'da strukturali log: `os.Logger` {#logger}

`print()` production'da yaramaydi: filtrlab bo'lmaydi, darajasi yo'q, maxfiy ma'lumotni ochiq yozadi. Apple'ning **unified logging** tizimi (`os.Logger`) esa log'ni subsystem/category bo'yicha ajratadi, darajalar beradi va maxfiylikni boshqaradi:

```swift
import Foundation
import os

enum Log {
    static let network = Logger(subsystem: "uz.azimjondev.app", category: "network")
    static let purchase = Logger(subsystem: "uz.azimjondev.app", category: "purchase")
}

func fetchProfile(userID: String) async {
    // Dinamik qiymatlar standart holatda maxfiy (<private>) yoziladi.
    Log.network.info("fetching profile for \(userID, privacy: .private)")
    do {
        let url = URL(string: "https://api.example.uz/profile")!
        let (_, response) = try await URLSession.shared.data(from: url)
        let status = (response as? HTTPURLResponse)?.statusCode ?? -1
        Log.network.debug("profile status \(status, privacy: .public)")
    } catch {
        Log.network.error("profile failed: \(error.localizedDescription, privacy: .public)")
    }
}
```

- Darajalar: `debug` → `info` → `notice` → `error` → `fault`. `debug` odatda faqat qurilmaga ulanganda ko'rinadi.
- Log'larni **Console.app**da subsystem bo'yicha filtrlab ko'rasiz.
- `privacy: .public` faqat maxfiy bo'lmagan qiymatlarga qo'yiladi (status kodi, xato matni). Email, ID, token — hech qachon `.public` emas.

---

## 3. Crash reporting va dSYM {#crash}

Crash bo'lganda qurilma **xotira manzillari**ni yozadi (`0x0000000104a3c2f8`). Ularni funksiya nomi va qator raqamiga aylantirish — **symbolication** — uchun **dSYM** fayli kerak. U archive paytida yaratiladi va crash xizmatiga yuklanishi shart.

|               | Firebase Crashlytics                           | Sentry                                                                    |
| ------------- | ---------------------------------------------- | ------------------------------------------------------------------------- |
| Narx          | Bepul                                          | Bepul tarif + pullik                                                      |
| dSYM yuklash  | Xcode Run Script (Firebase'ning `run` skripti) | `sentry-cli debug-files upload`                                           |
| Kuchli tomoni | Oddiy, Firebase ekotizimi bilan                | Ko'proq kontekst: breadcrumb, release health, performance, source context |

Sentry uchun CI qadami (`SENTRY_AUTH_TOKEN` secret sifatida):

```bash
#!/bin/bash
# Archive'dan keyin dSYM'larni Sentry'ga yuklash
set -euo pipefail
ARCHIVE_PATH="$RUNNER_TEMP/MyApp.xcarchive"

sentry-cli debug-files upload \
  --org my-org --project my-ios-app \
  --include-sources \
  "$ARCHIVE_PATH/dSYMs"
```

Crashlytics'ni Swift Package Manager orqali qo'shgan bo'lsangiz, Firebase hujjatidagi **Run Script** build phase'ini qo'shasiz. Uning aniq yo'li va "Input Files" ro'yxati [Firebase — Get readable crash reports](https://firebase.google.com/docs/crashlytics/ios/get-deobfuscated-reports)da.

Qoida: har bir **Release** build'ning dSYM'i yuklanganini CI'da tekshiring. Aks holda crash'lar "o'qib bo'lmaydigan" bo'lib qoladi.

---

## 4. MetricKit — Apple'ning bepul telemetriyasi {#metrickit}

**MetricKit** (iOS 13+) — tizim o'zi yig'adigan performance va diagnostika ma'lumotlari. Ilova ularni `MXMetricManager` orqali oladi, qo'shimcha SDK kerak emas:

- **`MXMetricPayload`** — odatda kuniga bir marta keladigan yig'ma metrikalar: ishga tushish vaqti, CPU, xotira, disk yozuvlari, hang vaqti, animatsiya "hitch"lari.
- **`MXDiagnosticPayload`** — alohida hodisalar: crash, hang, CPU exception, disk write exception. Har biri call stack bilan.

```swift
import MetricKit
import os

final class MetricsReporter: NSObject, MXMetricManagerSubscriber {
    private let logger = Logger(subsystem: "uz.azimjondev.app", category: "metrics")

    /// AppDelegate'da ilova ishga tushganda chaqiring.
    func start() {
        MXMetricManager.shared.add(self)
    }

    nonisolated func didReceive(_ payloads: [MXMetricPayload]) {
        for payload in payloads {
            // jsonRepresentation() — serveringizga yuborish uchun tayyor JSON
            let json = payload.jsonRepresentation()
            logger.info("metric payload: \(json.count) bytes")
        }
    }

    nonisolated func didReceive(_ payloads: [MXDiagnosticPayload]) {
        for payload in payloads {
            let crashes = payload.crashDiagnostics?.count ?? 0
            let hangs = payload.hangDiagnostics?.count ?? 0
            logger.error("diagnostics: crashes=\(crashes) hangs=\(hangs)")
        }
    }
}
```

Muhim: bu callback'lar **main thread'da emas**. Shu sababli metodlar `nonisolated` deb belgilangan (Swift 6 strict concurrency). UI'ni yangilashingiz kerak bo'lsa, `MainActor`ga o'ting.

Sentry ishlatsangiz, MetricKit'ni o'zingiz ulashingiz shart emas: Sentry Cocoa SDK **8.14.0+** (iOS 15+) da `options.enableMetricKit = true` sozlamasi bor. Manba: [Sentry — MetricKit](https://docs.sentry.io/platforms/apple/configuration/metric-kit/).

---

## 5. Xcode Organizer metrikalari {#organizer}

**Xcode → Window → Organizer → Metrics** bo'limi ma'lumot ulashishga rozi bo'lgan real foydalanuvchilardan yig'iladi. U yerda versiyalar bo'yicha quyidagilar ko'rinadi:

- **Launch Time** — ilovaning ishga tushish vaqti
- **Hang Rate** — UI "qotib qolish" chastotasi
- **Scrolling** — "hitch" (kadr tushishi)
- **Memory**, **Disk Writes**, **Battery Usage**
- **Terminations** — tizim ilovani nima sababdan yopgani

Yangi reliz chiqqach (ayniqsa phased release paytida — [DevOps 2](/ios-lessons/devops-release#phased)) shu grafiklarni oldingi versiya bilan solishtiring.

---

## 6. SLI, SLO va error budget {#slo}

- **SLI** (Service Level Indicator) — o'lchov. Masalan: _crash-free sessions_ ulushi.
- **SLO** (Service Level Objective) — maqsad. Masalan: _oylik crash-free sessions ≥ 99.9%_.
- **Error budget** — `1 − SLO`: shuncha "xato qilishga ruxsat" bor.

Hisoblaymiz: ilovada oyiga 1 200 000 sessiya bor, SLO = 99.9%.

```python
sessions_per_month = 1_200_000
slo = 0.999

budget_sessions = sessions_per_month * (1 - slo)
print(f"Oyiga ruxsat etilgan crash'li sessiyalar: {budget_sessions:.0f}")

# Xuddi shu 99.9% backend uptime uchun (30 kunlik oy):
minutes_per_month = 30 * 24 * 60
print(f"Ruxsat etilgan downtime: {minutes_per_month * (1 - slo):.1f} daqiqa")
```

Natija: oyiga **1 200** ta crash'li sessiya yoki backend uchun **43.2 daqiqa** downtime.

**Byudjet tugasa nima bo'ladi?** Jamoa yangi funksiyalarni to'xtatib, barqarorlikka o'tadi. Bu qoidani oldindan kelishib qo'yish kerak — aks holda SLO shunchaki "chiroyli raqam" bo'lib qoladi.

Mo'ljal uchun: vendor hisobotlariga ko'ra, iste'molchi ilovalar uchun crash-free sessions **≥ 99.5%**, fintech va sog'liq ilovalari uchun **≥ 99.9%** yaxshi daraja hisoblanadi ([Luciq 2026 benchmarks](https://www.luciq.ai/mobile-app-performance-benchmarks-2026)). Bu Apple'ning rasmiy standarti emas — o'z ilovangiz tarixiga qarab maqsad qo'ying.

---

## 7. Incident va blameless postmortem {#incident}

**Severity** (og'irlik darajasi) — muammoning ta'sirini tasvirlaydi, aybni emas:

| Daraja   | Misol                                                 | Javob                         |
| -------- | ----------------------------------------------------- | ----------------------------- |
| **SEV1** | To'lov ishlamayapti, ilova ochilishi bilan yiqilyapti | Darhol, hamma ishni to'xtatib |
| **SEV2** | Asosiy funksiya qisman ishlamayapti                   | Shu kun ichida                |
| **SEV3** | Kichik bug, aylanma yo'l bor                          | Keyingi reliz                 |

Mobil ilovadagi SEV1 uchun odatiy harakatlar ketma-ketligi: kill switch bilan funksiyani o'chirish → phased release'ni pauza qilish → hotfix → expedited review.

**Blameless postmortem** (Google SRE madaniyati) — "kim aybdor?" emas, "tizim nega buni o'tkazib yubordi?" degan savol. Hodisadan keyingi bir necha ish kuni ichida yoziladi. Shablon:

```text
Sarlavha: 2.4.0 — iOS 17'da ishga tushishda crash (SEV1)
Xulosa: 2 jumlada nima bo'ldi va ta'siri.
Ta'sir: qancha foydalanuvchi, qancha vaqt, qaysi versiyalar.
Timeline: 14:02 phased release 5% → 14:40 crash-free 97% → 14:55 pauza → ...
Sabablar: tizim darajasida (test matritsasida iOS 17 yo'q edi, alert chegarasi yo'q edi).
Nima yaxshi ishladi: kill switch 10 daqiqada ishga tushdi.
Action items: [egasi, muddat, bajarilganlik mezoni] har biri uchun.
```

Manba: [Google SRE Book — Postmortem Culture](https://sre.google/sre-book/postmortem-culture/).

---

## 8. Kichik backend uchun VPS: real konfiguratsiya {#vps}

Ko'p iOS dasturchilar ilova uchun kichik API (push, obuna tekshiruvi, admin panel) ham yuritadi. Quyidagi konfiguratsiyalar o'ylab topilmagan — ular **shu sayt (azimjondev.uz) serverida ishlab turgan** fayllarning soddalashtirilgan ko'rinishi: Node.js ilova + systemd + nginx + Let's Encrypt.

### SSH: faqat kalit bilan kirish

`/etc/ssh/sshd_config`dagi asosiy sozlamalar:

```text
PasswordAuthentication no
PermitRootLogin no
PubkeyAuthentication yes
```

> **Ogohlantirish:** parolni o'chirishdan oldin **boshqa terminal oynasida** kalit bilan kirish ishlashini tekshiring. Aks holda serverdan qulflanib qolasiz. O'zgartirishdan keyin: `sudo systemctl reload ssh`.

### Firewall (ufw)

```bash
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow OpenSSH
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable
sudo ufw status verbose
```

### systemd service — ilovani doim ishlab turadigan qilish

`/etc/systemd/system/myapi.service`:

```ini
[Unit]
Description=My API (Node/Express)
After=network.target

[Service]
Type=simple
User=www-data
Group=www-data
WorkingDirectory=/var/www/myapi
EnvironmentFile=/var/www/myapi/.env
Environment=NODE_ENV=production
Environment=PORT=3000
ExecStart=/usr/bin/node server.js
Restart=always
RestartSec=5

# Hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ReadWritePaths=/var/www/myapi/data

[Install]
WantedBy=multi-user.target
```

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now myapi
journalctl -u myapi -f   # log'larni jonli kuzatish
```

- `Restart=always` — jarayon yiqilsa, 5 soniyadan keyin qayta ko'tariladi.
- `EnvironmentFile` — secret'lar kod bilan birga emas, serverdagi alohida faylda turadi (`chmod 600`).
- Hardening qatorlari jarayonga faqat kerakli papkaga yozishga ruxsat beradi.

### nginx — reverse proxy

`/etc/nginx/sites-available/myapi`:

```nginx
server {
    listen 80;
    listen [::]:80;
    server_name api.example.uz;

    client_max_body_size 2m;

    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 30s;
    }

    # yashirin fayllar (.env, .git) hech qachon berilmaydi
    location ~ /\. {
        deny all;
    }
}
```

```bash
sudo ln -s /etc/nginx/sites-available/myapi /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx
```

`nginx -t` konfiguratsiyani tekshiradi. Xato bo'lsa, `reload` qilinmaydi va ishlab turgan sayt buzilmaydi.

### HTTPS — Let's Encrypt

```bash
sudo apt install -y certbot python3-certbot-nginx
sudo certbot --nginx -d api.example.uz
```

Certbot 443-port blokini va 80→443 redirect'ni o'zi qo'shadi, sertifikatni esa avtomatik yangilaydi.

### Docker qachon kerak?

Bitta server va bitta ilova uchun systemd + nginx yetarli va sodda. **Docker** muhitni har joyda bir xil qilish kerak bo'lganda foydali: bir nechta servis, jamoada turli OS, yoki ilovani boshqa serverga tez ko'chirish. Minimal **multi-stage** Dockerfile — birinchi bosqich build qiladi, ikkinchisiga faqat ishlash uchun kerakli fayllar o'tadi:

```dockerfile
FROM node:20-alpine AS build
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:20-alpine
WORKDIR /app
ENV NODE_ENV=production
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY --from=build /app/dist ./dist
USER node
EXPOSE 3000
CMD ["node", "dist/server.js"]
```

`package*.json` kod'dan **oldin** ko'chiriladi. Shunda dependency'lar o'zgarmasa, Docker `npm ci` qatlamini cache'dan oladi va build tezlashadi.

---

## 9. Secrets va CI {#secrets}

- Parol, token, `.p8` kalit — hech qachon repo'da emas (`.gitignore`da bo'lsa ham). Ular CI secret yoki serverdagi `EnvironmentFile`da turadi.
- Kalit sizib chiqqan bo'lishi mumkin bo'lsa — darhol **rotate** qiling: yangisini yarating, eskisini bekor qiling.
- Bulutga (AWS/GCP) deploy qilishda uzoq muddatli access key o'rniga GitHub Actions **OIDC**'dan foydalaning (`permissions: id-token: write`). Bunda har bir ish uchun qisqa muddatli token beriladi.

---

## 10. Xulosa {#xulosa}

- iOS'da log uchun `os.Logger`: darajalar, kategoriyalar, `privacy`. `print()` production'da emas.
- dSYM yuklanmasa, crash'lar o'qib bo'lmaydigan bo'ladi. Buni CI'da tekshiring.
- MetricKit va Xcode Organizer — bepul va rasmiy performance ma'lumoti.
- SLO + error budget "barqarorlikmi, yangi funksiyami?" degan bahsni raqam bilan hal qiladi.
- Kichik backend uchun systemd + nginx + certbot yetarli. `nginx -t` va `journalctl` — eng yaqin do'stlaringiz.

---

## Manbalar {#manbalar}

1. [Apple — Logging (os.Logger)](https://developer.apple.com/documentation/os/logging)
2. [Apple — MetricKit](https://developer.apple.com/documentation/metrickit)
3. [Sentry — MetricKit configuration](https://docs.sentry.io/platforms/apple/configuration/metric-kit/)
4. [Firebase — Get readable crash reports (dSYM)](https://firebase.google.com/docs/crashlytics/ios/get-deobfuscated-reports)
5. [Google SRE Book — Postmortem Culture](https://sre.google/sre-book/postmortem-culture/)
6. [Google SRE Book — Service Level Objectives](https://sre.google/sre-book/service-level-objectives/)
7. [Luciq — Mobile App Performance Benchmarks 2026](https://www.luciq.ai/mobile-app-performance-benchmarks-2026)
8. [Certbot — nginx on Ubuntu](https://certbot.eff.org/)