From 7f3236ea986932102c89ef86df2515602351e34f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=AA=20S=C4=A9=20B=C3=ADch?= Date: Fri, 18 Jun 2021 15:13:43 +0700 Subject: [PATCH] Notes for aws, rails encryption (#59) --- blog/2021-06-16-how-websocket-works.md | 62 +++ blog/2021-06-17-active-record-encryption.md | 289 ++++++++++++ .../auto-scaling-group.md | 39 +- books/aws-developer-associated/ec2.md | 427 +++++++++++++----- books/aws-developer-associated/elb.md | 106 +++-- books/aws-developer-associated/iam.md | 120 ++--- books/aws-developer-associated/intro.md | 12 +- .../security-group.md | 16 +- books/javascript/_category_.json | 3 + books/javascript/ecma.md | 27 ++ docs/ops/fluentd-logging.md | 174 ++++--- docs/ruby/action-text.md | 2 +- .../img/aws-developer-associated/iam-sts.webp | Bin 0 -> 11412 bytes static/img/blog/aes-block-cipher.png | Bin 0 -> 537728 bytes static/img/blog/aes-flow.jpeg | Bin 0 -> 25485 bytes static/img/blog/aes-round.jpeg | Bin 0 -> 12382 bytes static/img/blog/asymmetric-encryption.png | Bin 0 -> 143719 bytes static/img/blog/ctr-mode.png | Bin 0 -> 166305 bytes static/img/blog/gcm-mode.png | Bin 0 -> 39506 bytes static/img/blog/symmetric-encryption.png | Bin 0 -> 133781 bytes static/img/blog/websocket-handshake.jpeg | Bin 0 -> 32627 bytes 21 files changed, 986 insertions(+), 291 deletions(-) create mode 100644 blog/2021-06-16-how-websocket-works.md create mode 100644 blog/2021-06-17-active-record-encryption.md create mode 100644 books/javascript/_category_.json create mode 100644 books/javascript/ecma.md create mode 100644 static/img/aws-developer-associated/iam-sts.webp create mode 100644 static/img/blog/aes-block-cipher.png create mode 100644 static/img/blog/aes-flow.jpeg create mode 100644 static/img/blog/aes-round.jpeg create mode 100644 static/img/blog/asymmetric-encryption.png create mode 100644 static/img/blog/ctr-mode.png create mode 100644 static/img/blog/gcm-mode.png create mode 100644 static/img/blog/symmetric-encryption.png create mode 100644 static/img/blog/websocket-handshake.jpeg diff --git a/blog/2021-06-16-how-websocket-works.md b/blog/2021-06-16-how-websocket-works.md new file mode 100644 index 0000000..920cc0a --- /dev/null +++ b/blog/2021-06-16-how-websocket-works.md @@ -0,0 +1,62 @@ +--- +slug: websocket-hoat-dong-nhu-the-nao +title: WebSocket hoạt động như thế nào? +author: Lê Sĩ Bích +author_title: Ruby on Rails/React Developer +author_url: https://github.com/moonlight8978 +author_image_url: https://avatars.githubusercontent.com/u/26299310?v=4 +tags: [websocket, javascript, web] +--- + +## WebSocket là gì + +Cũng giống như HTTP, WebSocket là 1 protocol, hoạt động trên mô hình client/server, và sử dụng TCP connection. +Là một protocol, có nghĩa là bất kể việc implement ra sao, nhưng server cứ lắng nghe 1 TCP port, và giao tiếp giữa client/server tuân theo đúng spec mà WebSocket đề ra là được. + +WebSocket cho phép trao đổi dữ liệu 2 chiều giữa client và server. Tuy nhiên chắc sẽ có nhiều người thắc mắc + +> Sao không dùng HTTP rồi set interval request lên, hay là server push cho đơn giản. Đẻ ra lắm protocol làm gì đau đầu? + +Nếu sử dụng HTTP, mỗi lần cập nhật data mới thì client lại phải tạo request HTTP, và sẽ có rất nhiều header dư thừa, gây lãng phí băng thông, và việc khởi tạo request cũng mất thêm 1 chút thời gian. Trong khi đó WebSocket, sau khi client và server _say hello_ với nhau, TCP connection sẽ vẫn alive, và chúng sẽ giao lưu kết hợp với nhau qua connection đó mà không cần thêm những thông tin dư thừa như header. Và có lẽ còn nhiều lợi ích nữa, nhưng mình không biết =)) + +Khen thế đủ rồi, hãy thử tìm hiểu xem WebSocket ngang dọc ra sao. + +## Cách hoạt động của WebSocket + +### Tiền đề + +Ví dụ với ứng dụng mạng xã hội, ta có 1 tính năng chat. Mỗi khi bật chat với 1 người, 1 WebSocket tới webserver của ta sẽ được khởi tạo. Như vậy, ta đang có 1 webserver, và đang cần 1 websocket connection tới webserver đó. Mặt khác việc sử dụng raw TCP trên client cũng rất hạn chế. Chính vì vậy, WebSocket đã tận dụng luôn port 80/443 của HTTP(S) để khởi tạo connection. + +### Handshaking + +Khi khởi tạo connection, client sẽ tạo 1 HTTP request để thực hiện _handshake_ với server + +![handshake](/img/blog/websocket-handshake.jpeg) + +- Trước tiên client sẽ gửi request tới endpoint mà websocket server đang lắng nghe. ví dụ như `/chat`. + Request sẽ kèm theo 1 vài header mà WebSocket protocol quy định: `Sec-WebSocket-Key`, `Sec-WebSocket-Version`, `Upgrade`, `Connection`, ... và những header cơ bản của 1 HTTP request khác nữa. + +- WebSocket server sẽ tiếp nhận request, authen (nếu có), nếu không hợp lệ sẽ trả về `400 Bad Request` và terminate connection đó. + Ngược lại nếu OK, server sẽ trả về `101 Switching Protocols` cùng với header `Sec-WebSocket-Accept`. Connection đó sẽ được giữ, và sau đó client và server bắt đầu trao đổi thông tin qua lại với nhau. + +Giá trị của `Sec-WebSocket-Accept` được quy định như sau: + +```txt +Sec-WebSocket-Accept = Base64( SHA1 ( Sec-WebSocket-Key + 258EAFA5-E914-47DA-95CA-C5AB0DC85B11 ) ) +``` + +`258EAFA5-E914-47DA-95CA-C5AB0DC85B11` còn được gọi là _magic string_ + +### Trao đổi dữ liệu + +Việc trao đổi dữ liệu giữa client và server khá là hại não, cụ thể việc decode message, hay format message có thể xem thêm tại [đây](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_servers#exchanging_data_frames). Giữ chỗ để sau này nếu không ~~lười~~ bận quay lại tìm hiểu sau =)) + +Trong quá trình connection được keep alive, sẽ có rất nhiều request ping giữa client/server để kiểm tra xem phía bên kia còn active hay không. Bên gửi ping, bên nhận pong, vẫn còn sống thì ta lại giao lưu kết hợp tiếp. Còn không thì server/client sẽ biết để terminate inactive connection đó đi. + +## Tham khảo + +- https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_servers + +- https://stackoverflow.com/questions/50197453/how-long-can-a-websocket-connection-last + +- https://datatracker.ietf.org/doc/html/rfc6455 diff --git a/blog/2021-06-17-active-record-encryption.md b/blog/2021-06-17-active-record-encryption.md new file mode 100644 index 0000000..67d0994 --- /dev/null +++ b/blog/2021-06-17-active-record-encryption.md @@ -0,0 +1,289 @@ +--- +slug: tim-hieu-ve-active-record-encryption +title: Tìm hiểu về Active Record Encryption +author: Lê Sĩ Bích +author_title: Ruby on Rails/React Developer +author_url: https://github.com/moonlight8978 +author_image_url: https://avatars.githubusercontent.com/u/26299310?v=4 +tags: [websocket, javascript, web] +--- + +## Overview + +Vào ngày 28/5/2021, trên blog của Rails có thông báo về 1 tính năng mới đó là [ActiveRecord Encryption](https://weblog.rubyonrails.org/2021/5/28/this-week-in-rails-active-record-encrytion-several-performance-optimizations-and-much-more/). + +ActiveRecord Encryption giúp encrypt attribute nào đó của model, và lưu vào DB dưới dạng mã hoá. Tính năng này được extract ra từ dự án [HEY](https://www.hey.com) của Basecamp. + +Hãy thử tìm hiểu xem tính năng này có gì thú vị. + +## Usage + +Theo [guide](https://edgeguides.rubyonrails.org/active_record_encryption.html#basic-usage), đầu tiên chúng ta sẽ cần phải chạy câu lệnh sau + +```bash +rails db:encryption:init +``` + +Rails sẽ cho ta 1 gợi ý là copy đống sau vào credentials. Ừ `EDITOR=vim rails credentials:edit` rồi copy paste thôi. + +```yml +active_record_encryption: + primary_key: EGY8WhulUOXixybod7ZWwMIL68R9o5kC + deterministic_key: aPA5XyALhf75NNnMzaspW7akTfZp0lPY + key_derivation_salt: xEY0dt6TZcAMg52K7O84wYzkjvbA62Hz +``` + +Trong model, ta define attribute cần mã hoá như sau: + +```ruby title=app/models/article.rb +class Article < ApplicationRecord + encrypts :title +end +``` + +Rồi create/update model như bình thường + +```ruby +article = Article.create(title: "Encrypt it all!") +``` + +Và bùm, magic... + +```sql +INSERT INTO `articles` (`title`) VALUES ('{\"p\":\"n7J0/ol+a7DRMeaE\",\"h\":{\"iv\":\"DXZMDWUKfp3bg/Yu\",\"at\":\"X1/YjMHbHD4talgF9dt61A==\"}}') +``` + +Vô hạn bối rối... + +## AES + +Trước tiên ta hãy tìm hiểu về đống key mà Rails đã gen cho ta lúc đầu + +```yml +active_record_encryption: + primary_key: EGY8WhulUOXixybod7ZWwMIL68R9o5kC + deterministic_key: aPA5XyALhf75NNnMzaspW7akTfZp0lPY + key_derivation_salt: xEY0dt6TZcAMg52K7O84wYzkjvbA62Hz +``` + +Rails gen ra đống trên như nào vậy? + +```ruby title=https://github.com/rails/rails/blob/e2585a21e389967322a98b46a7b3643d82bd9c28/activerecord/lib/active_record/railties/databases.rake#L554-L566 +puts <<~MSG + Add this entry to the credentials of the target environment:#{' '} + + active_record_encryption: + primary_key: #{SecureRandom.alphanumeric(32)} + deterministic_key: #{SecureRandom.alphanumeric(32)} + key_derivation_salt: #{SecureRandom.alphanumeric(32)} +MSG +``` + +Nếu ai chưa biết thì đây là [standard lib của Ruby](https://ruby-doc.org/stdlib-3.0.1/libdoc/securerandom/rdoc/SecureRandom.html), đơn thuần là... random mà thôi =)) + +Thế đống key này để làm gì? Ở mục [Setup](https://edgeguides.rubyonrails.org/active_record_encryption.html#setup) của guide, Rails cũng đã gợi ý cho ta, đống này dùng cho AES. + +> this will be used to derive the AES 32 bytes key + +Vậy AES là cái khỉ mốc gì? Theo [wiki](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard), AES là viết tắt của **A**dvanced **E**ncryption **S**tandard. Hay còn được biết đến với cái tên Rijndael. AES là 1 chuẩn mã hoá, sử dụng symmetric encryption. AES rất phổ biến ngày nay. + +Thế bất nào mà lại đẻ ra thêm nhiều thuật ngữ hại não hơn vậy =)) Cứ từ từ rồi khoai sẽ nhừ, chúng ta sẽ tìm hiểu tiếp. + +Trước hết thì, thế nào là mã hoá? Ví dụ như ở 1 diễn đàn nào đó người ta share nhau hoàng thuỳ link bằng đống kí tự này + +> 68747470733A2F2F7777772E676F6F676C652E636F6D2E766E2F + +thay vì huỵch toẹt ra là một đường link ~~đồi truỵ~~ nào đó. Nhưng do việc giải mã quá đơn giản, cụ thể là dùng mã HEX, nên đây chỉ là encode/decode sang 1 dạng dữ liệu khác mà thôi, và việc chuyển đổi thường khá nhanh. Các bạn có thể vào đây [đây](https://www.convertstring.com/vi/EncodeDecode/HexDecode) để decode chuỗi huyền bí trên. + +Còn encrypt và decrypt dùng trong cryptography nó ở 1 tầm cao khác. Thông thường sẽ có thêm 1 key (hoặc 1 cặp key public/private). Khi đó input sẽ là key + data. Và output sẽ là 1 chuỗi nào đó, chuỗi này cực kì khó để giải mã nếu không nắm trong tay key. Thông thường thì phải cho chạy thuật toán vét cạn - tức là thử từng key một, và việc này cần đến hàng triệu năm đối với cả siêu máy tính. + +- Với thuật toán dùng 1 key cho cả việc encrypt lẫn decrypt thì sẽ được gọi là **symmetric encryption** + +![symmetric](/img/blog/symmetric-encryption.png) + +- Thuật toán dùng 1 cặp key, trong đó public key để mã hoá, còn private key để giải mã, được gọi là **asymmetric encryption** + +![asymmetric](/img/blog/asymmetric-encryption.png) + +[Image source](https://hackernoon.com/symmetric-and-asymmetric-encryption-5122f9ec65b1) + +Quay lại với AES, nó dùng symmetric, nên sẽ dùng 1 key cho cả việc mã hoá lẫn giải mã. AES key có độ dài là 128, 192, hoặc 256 bit. Tuỳ vào độ dài key nó sẽ có tên khác nhau AES-128, AES-192, AES-256. Key càng dài thì việc xử lý càng lâu, nhưng đổi lại càng bảo mật hơn. Mặc định Rails sử dụng key có độ dài 256 bit (32 bytes) + +```ruby title=https://github.com/rails/rails/blob/e2585a21e389967322a98b46a7b3643d82bd9c28/activerecord/lib/active_record/encryption/cipher/aes256_gcm.rb#L16 +CIPHER_TYPE = "aes-256-gcm" +``` + +Về việc làm thế nào để AES mã hoá/giải mã thì mình sẽ không đề cập tại đây ~~vì mình cũng không có hiểu mấy =))~~, nếu ai muốn tìm hiểu thêm có thể xem link [này](https://www.youtube.com/watch?v=lnKPoWZnNNM), hoặc gúc gồ ~~. Ở dưới là bản preview, trông cái thuật toán nó nôm na như sau + +AES sẽ chia nhỏ input thành từng block để xử lý + +![block-cipher](/img/blog/aes-block-cipher.png) + +![flow](/img/blog/aes-flow.jpeg) + +![round](/img/blog/aes-round.jpeg) + +[Image Source](https://www.tutorialspoint.com/cryptography/advanced_encryption_standard.htm) + +Cơ mà chỉ quan tâm là một chuỗi đầu vào + 1 key, đi qua đống black magic rồi trở thành 1 chuỗi bảo mật siêu hạng là quá đủ cho một cuộc tình rồi. + +À mà nhìn cái constant Rails define ở trên kia, lại có thêm cái gì ở cuối chuỗi thế nhỉ. `gcm`???? Bối rối again. + +## GCM + +GCM là viết tắt của **G**alois/**C**ounter **M**ode... Nghe xong hiểu chết liền .\_\_. Mò thử xem nào! + +Như tên gọi của nó, GCM là một _mode_ trong symmetric encryption, kết hợp của **Counter Mode** và **Galois Mode**, dùng cho mã hoá dạng khối (block). Vậy thì AES đi với GCM là chuẩn bài rồi nhỉ. + +Ngoài GCM ra còn các mode khác như CCM, SIV, ...vân vân mây mây. Các bạn có thể xem thêm tại [đây](https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation) + +Trước tiên ta hãy bắt đầu từ **mode**, từ này nằm trong **Mode of Operation**. Các mode sẽ được apply trên từng block data (như đã nói ở trên thì AES chia data thành các block nhỏ và mã hoá), giúp cho output của việc mã hoá block bảo mật hơn. + +Counter mode sử dụng 1 số integer làm counter, qua từng block counter sẽ tăng lên 1. Ở mỗi block, số này sẽ được mã hoá cùng với data để có được output bảo mật hơn. + +![counter-mode](/img/blog/ctr-mode.png) + +Galois Mode là một authentication mode. Galois Mode giúp đảm bảo rằng cipher output của ta không bị chỉnh sửa bởi một bên thứ 3. Hoạt động như nào thì mình chịu à =)) + +Kết hợp 2 mode trên và ta có GCM, giúp thuật toán mã hoá của ta bảo mật hơn, đồng thời đảm bảo được tính toàn vẹn của dữ liệu. + +![gcm-mode](/img/blog/gcm-mode.png) + +Bên trên là giải thích sơ qua về thuật toán mã hoá mà Rails sử dụng, nếu có sai sót thì cũng đừng gạch đá tội mình =)) + +## AES-256-GCM meets Rails + +### Encryption Key + +```yml +active_record_encryption: + primary_key: EGY8WhulUOXixybod7ZWwMIL68R9o5kC + deterministic_key: aPA5XyALhf75NNnMzaspW7akTfZp0lPY + key_derivation_salt: xEY0dt6TZcAMg52K7O84wYzkjvbA62Hz +``` + +Quay trở lại với đống key, hãy cùng mò code đáy bể xem chúng dẫn ta tới đâu :v + +```ruby +# https://github.com/rails/rails/blob/9c091b4fd378df515c4c31b85bb6a968463a1d82/activerecord/lib/active_record/railtie.rb#L313-L317 +ActiveRecord::Encryption.configure \ + primary_key: app.credentials.dig(:active_record_encryption, :primary_key), + deterministic_key: app.credentials.dig(:active_record_encryption, :deterministic_key), + key_derivation_salt: app.credentials.dig(:active_record_encryption, :key_derivation_salt), + **config.active_record.encryption + +# https://github.com/rails/rails/blob/9c091b4fd378df515c4c31b85bb6a968463a1d82/activerecord/lib/active_record/encryption/configurable.rb#L20-L33 +def configure(primary_key:, deterministic_key:, key_derivation_salt:, **properties) + config.primary_key = primary_key + config.deterministic_key = deterministic_key + config.key_derivation_salt = key_derivation_salt + + context.key_provider = ActiveRecord::Encryption::DerivedSecretKeyProvider.new(primary_key) +end + +# https://github.com/rails/rails/blob/9c091b4fd378df515c4c31b85bb6a968463a1d82/activerecord/lib/active_record/encryption/derived_secret_key_provider.rb#L7-L9 +def initialize(passwords) + super(Array(passwords).collect { |password| Key.derive_from(password) }) +end +``` + +Xem tiếp class `Key` có gì nào. + +```ruby +# https://github.com/rails/rails/blob/9c091b4fd378df515c4c31b85bb6a968463a1d82/activerecord/lib/active_record/encryption/key.rb#L10-L26 +class Key + def initialize(secret) + @secret = secret + @public_tags = Properties.new + end + + def self.derive_from(password) + secret = ActiveRecord::Encryption.key_generator.derive_key_from(password) + ActiveRecord::Encryption::Key.new(secret) + end +end + +# https://github.com/rails/rails/blob/9c091b4fd378df515c4c31b85bb6a968463a1d82/activerecord/lib/active_record/encryption/key_generator.rb#L32-L34 +def derive_key_from(password, length: key_length) + ActiveSupport::KeyGenerator.new(password).generate_key(ActiveRecord::Encryption.config.key_derivation_salt, length) +end +``` + +Như vậy, 2 trong số 3 thanh niên của ta đã bị lộ mặt: + +- `primary_key` đóng vai trò là `password` param của `ActiveSupport::KeyGenerator.new` +- `key_derivation_salt` đóng vai trò là tham số thứ nhất của `ActiveSupport::KeyGenerator#generate_key` + +Hãy check tiếp `ActiveSupport::KeyGenerator` + +```ruby +# https://github.com/rails/rails/blob/9c091b4fd378df515c4c31b85bb6a968463a1d82/activesupport/lib/active_support/key_generator.rb#L26-L41 +def initialize(secret, options = {}) + @secret = secret + @iterations = options[:iterations] || 2**16 + @hash_digest_class = options[:hash_digest_class] || self.class.hash_digest_class +end + +def generate_key(salt, key_size = 64) + OpenSSL::PKCS5.pbkdf2_hmac(@secret, salt, @iterations, key_size, @hash_digest_class.new) +end +``` + +Qua hết wrapper của Rails rồi, giờ ta phải mò vào docs của Ruby thôi. +Theo [docs](https://ruby-doc.org/stdlib-3.0.1/libdoc/openssl/rdoc/OpenSSL.html#module-OpenSSL-label-Encryption), docs lại bảo hàm này giờ đổi tên thành [`OpenSSL::KDF.pbkdf2_hmac`](https://ruby-doc.org/stdlib-3.0.1/libdoc/openssl/rdoc/OpenSSL/KDF.html#method-c-pbkdf2_hmac) mất rồi... Ờ thì lại mò vào xem. Nôm na hàm này sẽ tạo ra 1 cipher key dùng cho việc mã hoá. + +Tại đây, ta có thể kết luận key mã hoá mà Rails dùng có: + +- `passphrase`: `primary_key` +- `salt`: `key_derivation_salt` +- `iterations`: 2^16 +- `key_len`: độ dài key 32 bytes +- `digest`: hash algorithm SHA-1 + +Hãy cùng test qua cái key này xem + +```ruby title=irb +require 'openssl' + +cipher = OpenSSL::Cipher.new('aes-256-gcm') +cipher.encrypt +iv = cipher.random_iv + +pwd = 'EGY8WhulUOXixybod7ZWwMIL68R9o5kC' +salt = 'xEY0dt6TZcAMg52K7O84wYzkjvbA62Hz' +iter = 2**16 +key_len = OpenSSL::Cipher.new("aes-256-gcm").key_len +digest = OpenSSL::Digest.new('SHA1') + +key = OpenSSL::PKCS5.pbkdf2_hmac(pwd, salt, iter, key_len, digest) +cipher.key = key + +encrypted = cipher.update "hello" +encrypted << cipher.final # => "\xB2#\x10\xE7n" + +cipher.decrypt +cipher.iv = iv +decrypted = cipher.update(encrypted) # => "hello" +``` + +Sau bao nhiêu vất vả thì cũng tìm được cái key =)) Vậy quá trình mã hoá như thế nào? Lại mò tiếp .\_\_. + +### Encryption Process + +Hãy cùng bắt đầu từ việc khai báo trong model + +```ruby title=app/models/article.rb +class Article < ApplicationRecord + encrypts :title +end +``` + +Rồi cùng tìm xem `ApplicationRecord.encrypts` nó làm trò gì nào + +## Tham khảo + +- https://edgeguides.rubyonrails.org/active_record_encryption.html +- https://www.precisely.com/blog/data-security/aes-vs-rsa-encryption-differences +- https://www.youtube.com/watch?v=lnKPoWZnNNM +- https://www.youtube.com/watch?v=g_eY7JXOc8U +- https://www.includehelp.com/cryptography/mode-of-operation.aspx diff --git a/books/aws-developer-associated/auto-scaling-group.md b/books/aws-developer-associated/auto-scaling-group.md index 7238add..4e23dc1 100644 --- a/books/aws-developer-associated/auto-scaling-group.md +++ b/books/aws-developer-associated/auto-scaling-group.md @@ -3,38 +3,45 @@ title: Auto Scaling Group code: N/A --- -* Collection of EC2 instances that are treated as group for the purposes of automatic scaling & management +## Overview -#### Capacity settings +- Collection of EC2 instances that are treated as group for the purposes of automatic scaling & management + +## Capacity settings - Min/Max/Desired Capacity - Availability zone: >= 1 -#### Health Check Replacements +## Health Check Replacements - ASG will perform health check on EC2 instances. ASG will terminate unhealthy one and launch new instance to keep desired capacity match. - 2 types of check - EC2 health check: If the instance status is one of `stopping`, `stopped`, `shutting-down`, or `terminated`, it will be considered as unhealthy - ELB check: Use attached load balancer to perform health check on `running` instances -#### Scaling Policies +## Scaling Policies + +- Scale In (Remove instances) / Scale Out (Add more instances) +- Target Tracking Scaling Policy: -* Scale In (Remove instances) / Scale Out (Add more instances) -* Target Tracking Scaling Policy: - * Maintains a specific metric at a target value - * ALB Request Count per target, Average CPU, Average network In/Out - * eg. If average CPU exceeds 75% then add another server + - Maintains a specific metric at a target value + - ALB Request Count per target, Average CPU, Average network In/Out + - eg. If average CPU exceeds 75% then add another server -* Simple Scaling Policy (legacy, in favour of Step Scaling): scale when alarm (Cloudwatch alarm) is breached -* Step Scaling: scale when an alarm is breached, can escalates based on alarm value changing +- Simple Scaling Policy (legacy, in favour of Step Scaling): scale when alarm (Cloudwatch alarm) is breached +- Step Scaling: scale when an alarm is breached, can escalates based on alarm value changing -#### ELB Integration +## ELB Integration -* Classic Load Balancers are associated directly to ASG -* Application and Network Load Balancers are associated indirectly via their Target Groups +- Classic Load Balancers are associated directly to ASG +- Application and Network Load Balancers are associated indirectly via their Target Groups -#### Launch Configuration +## Launch Configuration - Template for ASG to launch an instance - EC2 settings (storage, instance type, role, ...) -- Launch Templates are Launch Configurations with Versioning \ No newline at end of file +- Launch Templates are Launch Configurations with Versioning + +## Scaling Cooldowns + +- Help to ensure the ASG doesn't launch or terminate instances before the previous scaling activity takes effect diff --git a/books/aws-developer-associated/ec2.md b/books/aws-developer-associated/ec2.md index 02ea676..8448981 100644 --- a/books/aws-developer-associated/ec2.md +++ b/books/aws-developer-associated/ec2.md @@ -1,121 +1,314 @@ --- -title: AWS EC2 +title: AWS Elastic Compute Cloud (EC2) code: N/A --- -* Elastic Compute Cloud -* General purpose instance types: -* Usecase: Web servers - * Mac: Mac machine to build/test/develop iOS app - * T4g: - - * Busrtable general purpose - * Arm-based AWS Graviton2 CPU (Custom built AWS Graviton2 Processor with 64-bit Arm Neoverse cores) - * Unlimited mode by default - * EBS-optimized by default - * Network bandwidth: Up to 5Gbps - * T3: - * Burstable general purpose - * 2.5 GHz Intel Xeon Scalable Processor - * Unlimited mode by default - * EBS optimized - * Network bandwidth: Up to 5Gbps - * AWS Nito System, Xeon => 30% price performance improvement over T2 instances - * T3a: - * Burstable general purpose - * AMD EPYC 7000 series processors with an all core turbo clock speed of 2.5 GHz - * Unlimited mode by default - * AWS Nitro System - * EBS optimized - * Network: Up to 5Gbps - * Deliver up to 10% cost savings over comparable instance types - * T2: - * Burstable performance - * High frequency Intel Xeon processors (3.0 ~ 3.3GHz) - * Standard mode by default - * Network (Low - Moderate performance) - * M6g: - * Arm-based AWS Graviton2 processors - * EBS-optimized by default - * deliver up to 40% better price/performance over current generation M5 instances and offer a balance of compute, memory, and networking resources - * AWS Nitro System - * EBS or NVMe SSDs - * Network: 10~15Gbps - * M5: - * Latest generation of General Purpose Instances - * Network: 10-25Gbps - * Up to 3.1 GHz Intel Xeon® Platinum 8175M processors with new Intel Advanced Vector Extension (AVX-512) instruction set - * EBS or NVMe SSDs - * AWS Nitro System - * M5a: - * General Purpose - * AMD EPYC 7000 series processors with an all core turbo clock speed of 2.5 GHz - * Network: 10-20Gbps - * EBS or NVMe SSDs - * AWS Nitro System - * M5n: - * 2nd generation Intel Xeon Scalable Processors (Cascade Lake) with a sustained all-core Turbo CPU frequency of 3.1 GHz and maximum single core turbo frequency of 3.5 GHz - * Support for the new Intel Vector Neural Network Instructions (AVX-512 VNNI) (for Machine Learning) - * Network: 25-100Gbps - * EBS or NVMe SSDs - * AWS Nitro System - * M5zn: - * 2nd Generation Intel Xeon Scalable Processors (Cascade Lake) with an all-core turbo frequency up to 4.5 GHz - * Network 25-100Gbps - * High single thread performance - * M4: - * 2.3 GHz Intel Xeon® E5-2686 v4 (Broadwell) processors or 2.4 GHz Intel Xeon® E5-2676 v3 (Haswell) processors - * EBS-optimized by default - * Network: Moderate - 25Gbps - * A1: - * Custom built AWS Graviton Processor with 64-bit Arm Neoverse cores - * Network: Up to 10Gbps - * EBS-optimized by default - * AWS Nitro System -* Compute optimized: C instances - * Usecase: scientific modeling, dedicated game servers, and server engines -* Memory optimized: R, X instances - * Usecase: real-time big data analytics, in-memory cache, in-memory databases -* Accelerated computing: P4, P3, P2, Inf1, G4dn, G4ad, G3, F1 - * Usecase: machine learning, speech recognization, compute finance, ... -* Storage optimized: I, D, H instances - * Usecase: NoSQL, in-memory or transaction databases, data warehouse, ... -* Burstable - * Burstable: - * Traditional Amazon EC2 instance types provide fixed CPU utilization, while burstable performance instances provide a baseline level of CPU utilization with the ability to burst CPU utilization above the baseline level. - * The baseline utilization and ability to burst are governed by CPU credits. - * If a burstable performance instance uses fewer CPU resources than is required for baseline utilization (such as when it is idle), the unspent CPU credits are accrued in the CPU credit balance. If a burstable performance instance needs to burst above the baseline utilization level, it spends the accrued credits. The more credits that a burstable performance instance has accrued, the more time it can burst beyond its baseline when more CPU utilization is needed. - * CPU credits: The following scenarios all use one CPU credit - * One vCPU at 100% utilization for one minute - * One vCPU at 50% utilization for two minutes - * Two vCPUs at 25% utilization for two minutes - * EBS-optimized: - * Unlimited mode: - * A burstable performance instance configured as `unlimited` can sustain high CPU utilization for any period of time whenever required. - * Charges might apply if your average utilization over a rolling 24-hour period exceeds the baseline utilization of the instance. -* All instances use ECC RAM - -* Instance Profile - * Recommend over embeded credentials - * IAM Policy ---> IAM Role ---> Instance profile <--- EC2 Instance -* Placement groups: - * Are logical placement - * Cluster: - * Pack instances together inside a AZ - * Low-latency for tightly coupled note-to-node communication - * Cannot be multi AZ - * Well suited for HPC applications - * Partition: - * Spread across logical partitions (inside a AZ ?) - * Each partition does not share the same hardware with each other (rack per partition) - * Well suited for distributed and replicated workloads - * Spread - * Each instance is placed on a different rack - * Critical instances should be keep seperated from each other - * Up to 7 instances - * Can be multi AZ -* User data: Is a script that run when lauching an instance - * `curl http://169.254.169.254/latest/user-data` - * User data is executed as `root` by default -* Metadata - * `curl http://169.254.169.254/latest/meta-data` \ No newline at end of file +## Overview + +- Elastic Compute Cloud → ECC → EC2 +- Things can be configured: + - CPU, RAM (All instances use ECC RAM), Network (Public IP, Network speed) + - Storage (SSD) + - Firewall (Security Group) + - Bootstrap script (User Data): + - Run at the instance first start. Once run, it never run again. + - Run with the root user + +## Instance types + +- Mac: Mac machine to build/test/develop iOS app + +- T4g: + + - Busrtable general purpose + - Arm-based AWS Graviton2 CPU (Custom built AWS Graviton2 Processor with 64-bit Arm Neoverse cores) + +- Unlimited mode by default + + - EBS-optimized by default + - Network bandwidth: Up to 5Gbps + +- T3: + + - Burstable general purpose + - 2.5 GHz Intel Xeon Scalable Processor + - Unlimited mode by default + - EBS optimized + - Network bandwidth: Up to 5Gbps + - AWS Nito System, Xeon => 30% price performance improvement over T2 instances + +- T3a: + + - Burstable general purpose + - AMD EPYC 7000 series processors with an all core turbo clock speed of 2.5 GHz + - Unlimited mode by default + - AWS Nitro System + - EBS optimized + - Network: Up to 5Gbps + - Deliver up to 10% cost savings over comparable instance types + +- T2: + + - Burstable performance + - High frequency Intel Xeon processors (3.0 ~ 3.3GHz) + - Standard mode by default + - Network (Low - Moderate performance) + +- M6g: + + - Arm-based AWS Graviton2 processors + - EBS-optimized by default + - deliver up to 40% better price/performance over current generation M5 instances and offer a balance of compute, memory, and networking resources + - AWS Nitro System + - EBS or NVMe SSDs + - Network: 10~15Gbps + +- M5: + + - Latest generation of General Purpose Instances + - Network: 10-25Gbps + - Up to 3.1 GHz Intel Xeon® Platinum 8175M processors with new Intel Advanced Vector Extension (AVX-512) instruction set + - EBS or NVMe SSDs + - AWS Nitro System + +- M5a: + + - General Purpose + - AMD EPYC 7000 series processors with an all core turbo clock speed of 2.5 GHz + - Network: 10-20Gbps + - EBS or NVMe SSDs + - AWS Nitro System + +- M5n: + + - 2nd generation Intel Xeon Scalable Processors (Cascade Lake) with a sustained all-core Turbo CPU frequency of 3.1 GHz and maximum single core turbo frequency of 3.5 GHz + - Support for the new Intel Vector Neural Network Instructions (AVX-512 VNNI) (for Machine Learning) + - Network: 25-100Gbps + - EBS or NVMe SSDs + - AWS Nitro System + +- M5zn: + + - 2nd Generation Intel Xeon Scalable Processors (Cascade Lake) with an all-core turbo frequency up to 4.5 GHz + - Network 25-100Gbps + - High single thread performance + +- M4: + + - 2.3 GHz Intel Xeon® E5-2686 v4 (Broadwell) processors or 2.4 GHz Intel Xeon® E5-2676 v3 (Haswell) processors + - EBS-optimized by default + - Network: Moderate - 25Gbps + +- A1: + + - Custom built AWS Graviton Processor with 64-bit Arm Neoverse cores + - Network: Up to 10Gbps + - EBS-optimized by default + - AWS Nitro System + +- Compute optimized: C instances + + - Usecase: scientific modeling, dedicated game servers, and server engines + +- Memory optimized: R, X instances + + - Usecase: real-time big data analytics, in-memory cache, in-memory databases + +- Accelerated computing: P4, P3, P2, Inf1, G4dn, G4ad, G3, F1 + + - Usecase: machine learning, speech recognization, compute finance, ... + +- Storage optimized: I, D, H instances + + - Usecase: NoSQL, in-memory or transaction databases, data warehouse, ... + +- Burstable + - Burstable: + - Traditional Amazon EC2 instance types provide fixed CPU utilization, while burstable performance instances provide a baseline level of CPU utilization with the ability to burst CPU utilization above the baseline level. + - The baseline utilization and ability to burst are governed by CPU credits. + - If a burstable performance instance uses fewer CPU resources than is required for baseline utilization (such as when it is idle), the unspent CPU credits are accrued in the CPU credit balance. If a burstable performance instance needs to burst above the baseline utilization level, it spends the accrued credits. The more credits that a burstable performance instance has accrued, the more time it can burst beyond its baseline when more CPU utilization is needed. + - CPU credits: The following scenarios all use one CPU credit + - One vCPU at 100% utilization for one minute + - One vCPU at 50% utilization for two minutes + - Two vCPUs at 25% utilization for two minutes + - EBS-optimized: + - Unlimited mode: + - A burstable performance instance configured as `unlimited` can sustain high CPU utilization for any period of time whenever required. + - Charges might apply if your average utilization over a rolling 24-hour period exceeds the baseline utilization of the instance. + +## Instance Profile + +- Recommend over embeded credentials +- IAM Policy ---> IAM Role ---> Instance profile <--- EC2 Instance + +## Placement groups + +- Are logical placement +- Cluster: + - Pack instances together inside a AZ + - Low-latency for tightly coupled note-to-node communication + - Cannot be multi AZ + - Well suited for HPC applications +- Partition: + - Spread across logical partitions (inside a AZ ?) + - Each partition does not share the same hardware with each other (rack per partition) + - Well suited for distributed and replicated workloads +- Spread + - Each instance is placed on a different rack + - Critical instances should be keep seperated from each other + - Up to 7 instances + - Can be multi AZ + +## User data + +- A script that run when lauching an instance (only first launch, restarting will not trigger the execution by default) + + - Can be configured to be executed on restart: [link](https://aws.amazon.com/premiumsupport/knowledge-center/execute-user-data-ec2/) (the example use mime-multi part file) + + ```txt + Content-Type: multipart/mixed; boundary="//" + MIME-Version: 1.0 + + --// + Content-Type: text/cloud-config; charset="us-ascii" + MIME-Version: 1.0 + Content-Transfer-Encoding: 7bit + Content-Disposition: attachment; filename="cloud-config.txt" + + #cloud-config + cloud_final_modules: + - [scripts-user, always] + + --// + Content-Type: text/x-shellscript; charset="us-ascii" + MIME-Version: 1.0 + Content-Transfer-Encoding: 7bit + Content-Disposition: attachment; filename="userdata.txt" + + #!/bin/bash + /bin/echo "Hello World" >> /tmp/testfile.txt + --// + ``` + +- User data is executed as `root` by default + +- Debug (on Amazon AMI Linux 2) + +```bash +tail -f /var/log/cloud-init-output.log +``` + +```log +Jun 10 09:38:21 cloud-init[3233]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [1] +Jun 10 09:38:21 cloud-init[3233]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts) +Jun 10 09:38:21 cloud-init[3233]: util.py[WARNING]: Running module scripts-user () failed +``` + +```log +Cloud-init v. 19.3-43.amzn2 finished at Thu, 10 Jun 2021 09:43:39 +0000. Datasource DataSourceEc2. Up 11.33 seconds +``` + +## Metadata + +```bash +curl http://169.254.169.254/latest/meta-data +``` + +## Purchase options + +- Standard (On-Demand) + - Short workload +- Reserved: + - Reserved Instances: Long workload (1 year or 3 years) + - Convertible Reserved Instances: Long workload with flexible instances (instance type is changable) + - Scheduled Reserved Instances: scheduled +- Spot instances + - Short workload + - Can lose instances (less reliable) +- Dedicated Hosts: + - Book entire server/placement. Full control +- Dedicated Instances: + +- EBS Volume is a network drive (attach through network, not physical drive) which can be attached to instances while they run + + - Has latency + - Can be detached and attached to another instance + +## Elastic Block Store (EBS) + +### EBS + +- Allow data persistence (after instance termination) +- EBS are bound to availability zone +- Capacity: size in GBs, IOPS (I/O per second) +- When launch an instance, root volume is deleted on termination by default (can be unchecked), other volumes are not +- Has limited performance +- Pay for provision + +#### Snapshot + +- A backup of EBS at a point of time +- Recommend to detach EBS before snapshot (not necessary) +- Can be copied across AZ and region + +#### AMI + +- Amazon Machine Images +- Pre-package softwares for faster boot, monitoring, ... + +#### Types + +- gp2/gp3 (SSD): General Purpose + - Cost effective, low latency + - 1GB - 1TB + - Limited performance: + - Small gp2 can burst to 3000 IOPS + - Size and IOPS are linked, max is 16000 IOPS + - gp3: Newer generation, IOPS and throughput can be increased independently +- io1/io2 (SSD): Low-latency, high-throughput workloads + - io2: + - Recommended. Same price as io1 but higher durability and IOPS + - Sub-milisecond latency + - Support EBS multi-attach +- st1 (HDD): Throughput optimized. Frequently accessed +- sc1 (HHD): Cold HDD. Less frequently accessed + +Only SSD-type EBS can be used as boot volume + +### EC2 Instance Store + +- High performance hardware disk +- Connect directly to the server (physical connection) +- Lose the storage when stop (ephemeral storage). Risk of data lost when hardware fails +- Usecase: Buffer, cache, temporary content, ... + +### Multi-attach + +- Attach the same EBS volume to multiple EC2 instances in the same AZ +- Only works on io1/io2 family volume +- Each instance has full read/write permission on the volume +- Filesystem must be cluster-aware (not XFS, EX4, ...) + +## Elastic File System (EFS) + +- Network file system that can be mounted on many EC2 instances + +:::caution +Compatible with Linux based AMI only +::: + +- Works on multi AZ +- Grow to petabyte-scale automatically +- Expensive (3x gp2). Pay for actual usage + +:::tip +EFS-IA (Infrequently Accessed) for cost savings +::: + +- NFSv4.1 protocol +- Use security group to control access to EFS +- Encryption at reset using KMS +- Performance: + - General Purpose: Latency-sensitive + - Max I/O: higher latency, throughput, highly parallel + - Bursting mode diff --git a/books/aws-developer-associated/elb.md b/books/aws-developer-associated/elb.md index bd140ff..1b81151 100644 --- a/books/aws-developer-associated/elb.md +++ b/books/aws-developer-associated/elb.md @@ -3,15 +3,16 @@ title: Elastic Load Balancer code: N/A --- -#### Overview +## Overview - Can be physical hardware or virtual software that accepts incoming traffic, and then distribute the traffic to multiple targets - 3 type of ELB - Application Load Balancer (HTTP/HTTPS) - Network Load Balancer (TCP/UDP) - Classic Load Balancer (Legacy) +- Can be scaled but not instantaneous → Contact AWS for a _warm-up_ -#### Rules of Traffic +## Rules of Traffic - Listeners @@ -28,70 +29,101 @@ code: N/A - Target group includes: - Machines which can be matched using - * Instance (AWS-provisioned machines) - * IP address (on-premise machines) - * Lambda function + - Instance (AWS-provisioned machines) + - IP address (on-premise machines) + - Lambda function - Port on each machine (use for traffic forwarding and healthcheck) - Health check rule (can use path without ALB rule path prefix): perform on each instance - * ALB rules: `/devices*` forward to target group - * Target group healthcheck: `/` + - ALB rules: `/devices*` forward to target group + - Target group healthcheck: `/` - Healthcheck fails will not affect instance state, ELB will not route the traffic to that instance - For CLB, traffic is sent to Listeners, then it forwards the traffic to any registered EC2 instances. No rules is applied -#### Application Load Balancer +## Application Load Balancer -* To load balance HTTP/HTTPS traffic (Layer 7 load balancer) +- To load balance HTTP/HTTPS traffic (Layer 7 load balancer) -* Create ALB: +- Create ALB: - * Type: internal / or public (internet facing) + - Type: internal / or public (internet facing) - * Listener: Ports/Protocol which ALB listens to - * Can listen to multiple ports - * Availability zones - * Routing: `@ref` [Target group](#Target group) + - Listener: Ports/Protocol which ALB listens to + - Can listen to multiple ports + - Availability zones + - Routing: `@ref` [Target group](#Target group) + +- Route tables support: + + - URL (path) + - Hostname (`HOST` header) + - Query params + +- Map dynamic port (ECS) ![](https://images.viblo.asia/6b35f0c0-3ce6-46c1-9d9d-b3fd4ffe8fd8.jpg) -* Rules: guide ALB to route traffic to correct resources - * Ordering - Matcher - Processor - * Matcher: by header, by path, ... - * Processor: - * Forwarding: to some target groups - * Redirect: redirect to another url +- Rules: guide ALB to route traffic to correct resources + + - Ordering - Matcher - Processor + - Matcher: by header, by path, ... + - Processor: + - Forwarding: to some target groups + - Redirect: redirect to another url -#### Network Load Balancer +- Headers: + - `X-Forwarded-For`: real request IP + - `X-Forwarded-Port` + - `X-Forwarded-Proto`: protocol talks to load balancer + +## Network Load Balancer - Handle TCP/UDP traffic (Layer 4 LB) - Cross-zone Load Balancing - Suitable for Multiplayer Game, or when network performance is critical -#### Classic Load Balancer +## Classic Load Balancer -* Can balance HTTP/HTTPS (Layer 7) or TCP traffic (Layer 4) (not at the same time) -* Cross-zone Load Balancing -* 504 response is returned if the underlying application is not responding -* Deprecated +- Can balance HTTP/HTTPS (Layer 7) or TCP traffic (Layer 4) (not at the same time) +- Cross-zone Load Balancing +- 504 response is returned if the underlying application is not responding +- Deprecated -#### Sticky sessions +## Sticky sessions -* Use cookie (Layer 7) +- Use cookie (Layer 7) -* Advanced load balancing method that allows us to bind a user's session to a specific EC2 instance -* Ensure all requests from that session are sent to the same instance -* Typically utilized with a CLB -* Can be enabled for ALB, but can only be set on a Target Group instead of individual EC2 instances +- Advanced load balancing method that allows us to bind a user's session to a specific EC2 instance +- Ensure all requests from that session are sent to the same instance +- Typically utilized with a CLB +- Can be enabled for ALB, but can only be set on a Target Group instead of individual EC2 instances -* Useful when specific information is only stored locally on a single instance (stateful app) +- Useful when specific information is only stored locally on a single instance (stateful app) -#### X-Forwarded-For (XFF) header +## X-Forwarded-For (XFF) header - Represent user IPv4 address -#### Cross-Zone Load Balancing +## Cross-Zone Load Balancing - Normally, the load balancer route traffic to the targets in the same AZ - At least 2 subnet (only 1 subnet per AZ) => 2 AZ must be choosen to increase the availability of the LB - When Cross-Zone Load Balancing is enabled (on CLB or NLB), the traffic will be distrubuted evenly across all AZ -- \ No newline at end of file + +## Server Name Indication (SNI) + +- Solve the problem of loading multiple certificates onto single webserver +- SNI is a protocol, requires the client to indicate the hostname of the target server in the initial SSL handshake. The server will find the correct certificate or return the default one. +- Only supported by ALB and NLB + +## Connection draining + +- Naming: + + - CLB: connection draining + - ALB/NLB: deregistration delay + +- Is the time to complete the in-flight to the target before being deregistered + + - Default 300s + - ALB will stop sending request to the instances which in draining mode diff --git a/books/aws-developer-associated/iam.md b/books/aws-developer-associated/iam.md index 37d29c8..838c4c2 100644 --- a/books/aws-developer-associated/iam.md +++ b/books/aws-developer-associated/iam.md @@ -3,12 +3,23 @@ title: IAM code: N/A --- -#### Core +## Core -* User can be group into Group, which will have same permissions, make adding or removing permission easier -* Roles can have many policies attached. Various AWS resources alllow roles to be attached directly to them +- User can be group into Group (not required) -#### Policy + Groups only contain users, not other groups + + User can belong to multiple groups + + All users in a group share the same permissions, make adding or removing permission easier (except inline policies) + +- Permission is granted by using policies (JSON document) + + Least privilege principle + +- Roles can have many policies attached. Various AWS resources alllow roles to be attached directly to them + +## Policy - Types - Managed Policies: Managed by AWS, not editable @@ -16,73 +27,74 @@ code: N/A - Inline Policies: Policies which directly attached to user - Structure - Statement: Array of policies - - Sid: Label + - Sid: Identifier of statement - Effect: Allow/Deny - - Principal: account, user, role, or federated user to allow or deny access + - Principal: account, user, role, or federated user to which the policy applied to - Action: list of actions to deny/allow - - Resource: The resource to which the actions applies - - Condition: Circumstances under which the policy grants permission + - Resource: list of resources to which the actions applied to + - Condition: Condition for this policy is in effect + +## User -#### User +- Password policy: to set minimum requirements to user password +- Programmatic access key +- Multi-factor authentication (MFA) -* Password policy: to set minimum requirements to user password -* Programmatic access key -* Multi-factor authentication (MFA) - * Cannot be forced to enabled - * Admin can create policy requiring MFA to access resources + - Cannot be forced to enabled + - Admin can create policy requiring MFA to access resources -* Temporary Security Credentials +- Temporary Security Credentials - * Like programmatic access keys, except they are temporary - * Useful in scenarios that involve: identity federation, delegation, cross-account access, IAM roles - * Can last from minutes to hours + - Like programmatic access keys, except they are temporary + - Useful in scenarios that involve: identity federation, delegation, cross-account access, IAM roles + - Can last from minutes to hours - * Not stored with the user, are generated dynamically and provided to the user when requested - * Basis for roles and identity federation + - Not stored with the user, are generated dynamically and provided to the user when requested + - Basis for roles and identity federation -### Role +## Role -* Based on Temporary Security Credentials -* Types: - * AWS Service Role: a role that a service assumes to perform actions in AWS account - * AWS service role for EC2 instance - * Special type of service role for that an application running on EC2 instance - * AWS service-linked role - * A unique type of service role that is linked directly to an AWS service. - * Are predefined, and includes all the permissions that the service requires -* Delegation: See **Cross-account roles** below +- Based on Temporary Security Credentials +- Types: + - AWS Service Role: a role that a service assumes to perform actions in AWS account + - AWS service role for EC2 instance + - Special type of service role for that an application running on EC2 instance + - AWS service-linked role + - A unique type of service role that is linked directly to an AWS service. + - Are predefined, and includes all the permissions that the service requires +- Delegation: See [Cross-account roles](#cross-account-roles) below -#### Identity Federation +## Identity Federation -* Linking multiple identity management systems -* Support: - * Enterprise: SAML, Custom Federation broker - * Web identity federation: Amazon, Facebook, Google, OpenID Connect 2.0 +- Linking multiple identity management systems +- Support: + - Enterprise: SAML, Custom Federation broker + - Web identity federation: Amazon, Facebook, Google, OpenID Connect 2.0 -#### Security Token Service (STS) +## Security Token Service (STS) -* STS is a web service that enables us to request temporary, limited-privilege credentials for IAM users or federated users +- STS is a web service that enables us to request temporary, limited-privilege credentials for IAM users or federated users -* Global service +- Global service -* All requests go to a single endpoint at https://sts.amazonaws.com +- All requests go to a single endpoint at https://sts.amazonaws.com - * Response include: AccessKeyID, SecretAccessKey, SessionToken, Expiration - * Some API actions to obtain STS: - * AssumeRole - * AssumeRoleWithSAML - * AssumeRoleWithWebIdentity - * ... + - Response include: AccessKeyID, SecretAccessKey, SessionToken, Expiration + - Some API actions to obtain STS: + - AssumeRole + - AssumeRoleWithSAML + - AssumeRoleWithWebIdentity + - ... -* AssumeRoleWithWebIdentity +- AssumeRoleWithWebIdentity - ![](https://images.viblo.asia/55adfd7f-f755-42fd-a17f-27f13dd82378.png) + ![](/img/aws-developer-associated/iam-sts.webp) -#### Cross-account roles +## Cross-account roles -* Grant users from another +- Grant users from another -* Policy +- Policy ```json { @@ -94,8 +106,12 @@ code: N/A } ``` -#### Practical notes +## Audit + +- IAM Credentials Report: List all users, and their status +- IAM Access Advisor: Show service permissions granted to a user and when those services were last accessed + +## Practical notes - Use groups to assign permissions to IAM users - Use IAM role instead of private keys - diff --git a/books/aws-developer-associated/intro.md b/books/aws-developer-associated/intro.md index 1653af7..5c262df 100644 --- a/books/aws-developer-associated/intro.md +++ b/books/aws-developer-associated/intro.md @@ -4,6 +4,12 @@ sidebar_position: 1 # Intro +## Course + +- https://www.udemy.com/course/aws-certified-developer-associate-dva-c01/ + +- https://www.youtube.com/watch?v=RrKRN9zRBWs + ## Break down | Domain | Rate | Questions | @@ -14,18 +20,18 @@ sidebar_position: 1 | Refactoring | 10% | 6.5 | | Monitoring & troubleshooting | 12% | 7.8 | -#### Summary +## Summary - https://github.com/gitvani/aws-mindmap - http://mistwire.com/2016/05/aws-certified-solutions-architect-associate-study-notes/ -#### Encryption +## Encryption - https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingEncryption.html - https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.Encryption.html - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html -#### EC2 +## EC2 - EC2 Instance Types https://aws.amazon.com/ec2/instance-types/ - Instance Store vs EBS https://aws.amazon.com/premiumsupport/knowledge-center/instance-store-vs-ebs/ diff --git a/books/aws-developer-associated/security-group.md b/books/aws-developer-associated/security-group.md index f0d8f06..ea226a3 100644 --- a/books/aws-developer-associated/security-group.md +++ b/books/aws-developer-associated/security-group.md @@ -1,7 +1,19 @@ --- title: AWS Security Group -code: N/A --- -* +## Overview +- Acting as a firewall +- Deny all traffic by default, only contain _Allow_ rules +- Source/Destination can be IP, IP range, another security group, or prefix list +- Association with instance: n-n + +## Popular classic ports + +- SSH (log onto a Linux instance): 22 +- SFTP - Secure File Transfer Protocol: 22 +- FTP: 21 +- HTTP: 80 +- HTTPS: 443 +- RDP (log onto a Windows instance): 3389 diff --git a/books/javascript/_category_.json b/books/javascript/_category_.json new file mode 100644 index 0000000..2937e90 --- /dev/null +++ b/books/javascript/_category_.json @@ -0,0 +1,3 @@ +{ + "label": "Javascript" +} diff --git a/books/javascript/ecma.md b/books/javascript/ecma.md new file mode 100644 index 0000000..67437b1 --- /dev/null +++ b/books/javascript/ecma.md @@ -0,0 +1,27 @@ +# ECMA + +## ECMA Stage + +- Stage 0 **Strawperson** + +> initial idea + +- Stage 1 **Proposal** + +> formal proposal that describes a discrete problem or general need, suggests a shape of the solution and points out potential challenges + +- Stage 2 **Draft** + +> The draft should describe the syntax, semantics and APIs precisely as much as possible. +> +> An experimental implementation is also needed, runnable by a browser or a build-time transpiler like Babel. + +- Stage 3 **Candidate** + +> almost final - but ready for feedback and refinements from implementations and users + +- Stage 4 **Finished** + +> the proposal is ready to be included in the latest draft of the specification - and be delivered with its next edition + +[Source](https://tc39.es/process-document/) diff --git a/docs/ops/fluentd-logging.md b/docs/ops/fluentd-logging.md index 1da24b5..caa2c78 100644 --- a/docs/ops/fluentd-logging.md +++ b/docs/ops/fluentd-logging.md @@ -1,28 +1,26 @@ # Logging with Fluentd, Logrorate -> Last updated: 2021-02-05 - ## No extra server required ### Overview -* Basic flow: Input => Filter => Output +- Basic flow: Input => Filter => Output - * Input: + - Input: - * How to get log - * Listen on UDP traffic - * Tail a system file - * `parser` can be applied to get right log format + - How to get log + - Listen on UDP traffic + - Tail a system file + - `parser` can be applied to get right log format - * Filter: + - Filter: - * Apply transformation to event + - Apply transformation to event - * Add custom attribute - * Parse log file + - Add custom attribute + - Parse log file - * Raw log go through `parser` will become `log event` + - Raw log go through `parser` will become `log event` ```txt tag: app.event @@ -30,23 +28,23 @@ record: {"host":"192.168.0.1","size":777,"method":"PUT"} ``` - * Output: + - Output: - * How to store/forward the log event - * Save in elasticsearch - * Save to file - * Save to S3 - * Each output has default `formatter` to format log event to appropriate format - * `out_file` has `out_file` formatter as default which will transform log event into `