Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sqlite and MySql support for asynk #141

Merged
merged 91 commits into from
Apr 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
91 commits
Select commit Hold shift + click to select a range
2eb4b48
possible impl with sqlx, we need to refactor dependencies and feature…
pxp9 Aug 26, 2023
1a09371
fix clippy xd
pxp9 Aug 26, 2023
cbe2d99
finally sqlite almost work, need help to debug something
pxp9 Aug 28, 2023
63b39c3
deleting some expects
pxp9 Aug 28, 2023
fb7d4ba
sqlite may work in workflow
pxp9 Aug 28, 2023
2ff47e1
stupid workflow work
pxp9 Aug 29, 2023
fdbe924
plz work
pxp9 Aug 29, 2023
20f376a
plz work , think i have fixed
pxp9 Aug 29, 2023
e79c0ee
fix workflow , now will work :D
pxp9 Aug 29, 2023
7e1ac88
this should work
Dopplerian Aug 29, 2023
4f308f2
this will definitely work
Dopplerian Aug 29, 2023
204a38a
clean up so it works incorrectly faster
Dopplerian Aug 29, 2023
fdb4f1e
okay that didn't work so idk what i'm doing here
Dopplerian Aug 29, 2023
5ba9910
if this works i'm dropping out of university
Dopplerian Aug 29, 2023
5119dcf
store uuid as text , so this way `PostgreSQL` schema will not change
pxp9 Aug 29, 2023
168b476
fix comment
pxp9 Aug 29, 2023
f4459d9
fix clippy
pxp9 Aug 29, 2023
bf53cfa
implement backend sqlx trait that allows better code
pxp9 Aug 29, 2023
0040820
better implementation with enum variants
pxp9 Aug 30, 2023
d6eb6e5
Merge branch 'master' into support-sqlite-mysql-asynk
Dopplerian Aug 31, 2023
08238bc
`Makefile` now creates `tests_sqlite` directory
Dopplerian Aug 31, 2023
169d955
make `asynk` tests use `.env` variables
Dopplerian Aug 31, 2023
4589673
drop unnecessary `drop`
Dopplerian Aug 31, 2023
a11ef55
query types as enum
pxp9 Sep 1, 2023
3332c83
fix clippy
pxp9 Sep 1, 2023
5f1c601
fix fmt, my bad i had a missconf in my home computer xd
pxp9 Sep 1, 2023
67bf9d0
stupid mysql does not work anything , i hate it so much :/
pxp9 Sep 1, 2023
fa4f135
rework to make it work with MySQL
pxp9 Sep 2, 2023
16e94ef
debugging issue : the issue is related to how a uniq task is inserted…
pxp9 Sep 2, 2023
eaa7bc0
MySQL own mutex , why Mysql is so gae ?
pxp9 Sep 2, 2023
f3a14d4
fix clippy
ayrat555 Sep 8, 2023
c57b0ba
split test jobs
ayrat555 Sep 8, 2023
b5deaed
fix build
ayrat555 Sep 8, 2023
9ffbf51
fix github actions
ayrat555 Sep 8, 2023
f8a87fd
do not run github actions twice
ayrat555 Sep 8, 2023
5592b09
fix formatting build
ayrat555 Sep 8, 2023
1bbe5ef
another approach
ayrat555 Sep 8, 2023
0501322
tokio console debug
pxp9 Apr 8, 2024
56c2dc9
fixing workflow
pxp9 Apr 8, 2024
c35cc3b
fixing workflow again
pxp9 Apr 8, 2024
218dd3b
i think i finally fixed
pxp9 Apr 8, 2024
f09c8e6
fixed \!
pxp9 Apr 8, 2024
8dd0c89
fixed \?
pxp9 Apr 8, 2024
c17740b
i am stupid now should be fixed :)
pxp9 Apr 8, 2024
d6f95fc
test blocking and fang derive error
pxp9 Apr 8, 2024
847a281
add postgres service to blocking workflow
pxp9 Apr 8, 2024
8230d9a
running async worker tests and blocking worker tests
pxp9 Apr 8, 2024
0a23009
cargo term color sqlite
pxp9 Apr 8, 2024
cc792ab
change sqlx version to 0.6.3 and using Pool instead of Transactions
pxp9 Apr 9, 2024
262d5d6
fix workflow for mysql
pxp9 Apr 9, 2024
41584c7
fix workflow
pxp9 Apr 9, 2024
4b0bdad
fix clippy, warns and mysql workflow
pxp9 Apr 9, 2024
192722d
adress comments and fix workflow
pxp9 Apr 9, 2024
db93dbb
fix mysql healthcheck
pxp9 Apr 9, 2024
32905b2
fix mysql healthcheck, yeap again
pxp9 Apr 9, 2024
2b30c0a
remove healthcheck
pxp9 Apr 9, 2024
9f0c74d
healthcheck is needed , plz work
pxp9 Apr 9, 2024
5ed857f
plz just work :(
pxp9 Apr 9, 2024
330539a
try to fix workflow
ayrat555 Apr 12, 2024
db904e6
update workflow to use a non deprecated version
pxp9 Apr 12, 2024
e31296d
return an error instead of panicking with fock
pxp9 Apr 12, 2024
6f85163
deleting prints and updating comments
pxp9 Apr 12, 2024
a468938
deleting unwraps or justifying them
pxp9 Apr 12, 2024
27ea179
README and moving each backend to separate files
pxp9 Apr 12, 2024
46540db
final README
pxp9 Apr 12, 2024
665ac47
bump fang version and bump rust version
pxp9 Apr 12, 2024
bd1e1a8
nit: added missing newline for consistency
Dopplerian Apr 13, 2024
a291459
update readme
Dopplerian Apr 13, 2024
108d55c
bump fang version following semver
Dopplerian Apr 13, 2024
0e59175
address ayrat issues
pxp9 Apr 14, 2024
7ec0e79
Fix/independent decoding (#149)
pxp9 Apr 15, 2024
890531a
relocate comment
Dopplerian Apr 15, 2024
fe69c66
remove `backend` field from `AsyncQueue`
Dopplerian Apr 15, 2024
201bc3b
fix clippy
Dopplerian Apr 15, 2024
9f540b7
worst clippy fix of all time
Dopplerian Apr 15, 2024
0dd461f
fmt fix
Dopplerian Apr 15, 2024
edf744a
use variant NoBackend instead Option
pxp9 Apr 15, 2024
f0ae9fe
fix clippy
pxp9 Apr 15, 2024
ce47e61
delete dummy variant
pxp9 Apr 15, 2024
baef61e
changing sqlite and mysql migrations
pxp9 Apr 15, 2024
527868b
sqlx uuid encoding and decoding for sqlite and postgres
pxp9 Apr 16, 2024
0f0be07
delete uuid previous impl comment
pxp9 Apr 16, 2024
c611d60
fix retry_task impl
pxp9 Apr 16, 2024
45d896a
fix mysql timestamp precision
pxp9 Apr 17, 2024
ea85604
rework encoding Datime<Utc> for postgresql and sqlite
pxp9 Apr 17, 2024
94fd332
rework encoding Uuid for mysql
pxp9 Apr 17, 2024
9da8548
fix clippy
pxp9 Apr 17, 2024
aafb352
fix serde_json::Value decoding and encoding
pxp9 Apr 17, 2024
e7c15bb
README stuff
pxp9 Apr 17, 2024
af8e0b4
improving API
pxp9 Apr 18, 2024
adff5b2
improving worker API
pxp9 Apr 18, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions .env
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,11 @@ SQLITE_FILE=fang.db
SQLITE_DIESEL_DIR=fang/sqlite_migrations
SQLITE_MIGRATIONS=${SQLITE_DIESEL_DIR}/migrations
SQLITE_CONFIG=${SQLITE_DIESEL_DIR}/diesel.toml
SQLITE_TESTS_DIR=tests_sqlite

HOST=127.0.0.1
POSTGRES_URL=postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${HOST}/${POSTGRES_DB}
MYSQL_URL=mysql://${MYSQL_USER}:${MYSQL_PASSWORD}@${HOST}/${MYSQL_DB}
POSTGRES_BASE_URL=postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${HOST}
POSTGRES_URL=${POSTGRES_BASE_URL}/${POSTGRES_DB}
MYSQL_BASE_URL=mysql://${MYSQL_USER}:${MYSQL_PASSWORD}@${HOST}
MYSQL_URL=${MYSQL_BASE_URL}/${MYSQL_DB}
DATABASE_URL=${POSTGRES_URL}
246 changes: 199 additions & 47 deletions .github/workflows/rust.yml
pxp9 marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,40 @@ name: Test and Build Rust
on:
push:
pull_request:
types: [opened, reopened]
schedule:
# Check if it works with current dependencies (weekly on Wednesday 2:32 UTC)
- cron: '32 2 * * 3'

env :
DATABASE_URL : postgres://postgres:postgres@localhost/fang

jobs:
test:
name: Test
clippy:
name: Clippy
runs-on: ubuntu-latest
env:
CARGO_TERM_COLOR: always

strategy:
matrix:
toolchain:
- stable

steps:
- uses: actions/checkout@v4


- name: Setup Rust
run: rustup update ${{ matrix.toolchain }} && rustup default ${{ matrix.toolchain }}


- name: Run clippy
run: cargo clippy --verbose --all-targets --all-features -- -D warnings

test_postgres_blocking:
name: Test blocking
runs-on: ubuntu-latest
env:
CARGO_TERM_COLOR: always

services:
# Label used to access the service container
Expand All @@ -35,67 +58,196 @@ jobs:
--health-timeout 5s
--health-retries 5

strategy:
matrix:
toolchain:
- stable

steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- name: Setup Rust
uses: actions-rs/toolchain@v1
with:
components: clippy
override: true
profile: minimal
toolchain: stable

- name: Run clippy
uses: actions-rs/cargo@v1
with:
command: clippy
args: --verbose --all-targets --all-features -- -D warnings
run: rustup update ${{ matrix.toolchain }} && rustup default ${{ matrix.toolchain }}

- name: Install diesel-cli
uses: actions-rs/cargo@v1
with:
command: install
args: diesel_cli --no-default-features --features "postgres"
run: cargo install diesel_cli --no-default-features --features postgres

- name: Setup Postgres db
working-directory: ./fang/postgres_migrations
run: diesel setup
run: diesel setup --database-url "postgres://postgres:postgres@localhost/fang"

- name: Run blocking tests
run: cargo test "blocking::queue::postgres" --verbose --features blocking --color always -- --nocapture

- name: Run blocking dirty tests
run: cargo test "blocking::worker" --verbose --features blocking -- --ignored

test_fang_derive_error:
name: Test fang_derive_error
runs-on: ubuntu-latest
env:
CARGO_TERM_COLOR: always

strategy:
matrix:
toolchain:
- stable

steps:
- uses: actions/checkout@v4

- name: Setup Rust
run: rustup update ${{ matrix.toolchain }} && rustup default ${{ matrix.toolchain }}

- name: Run fang derive error tests
run: cargo test "fang_derive_error" --verbose --color always -- --nocapture

test_postgres:
name: Test postgres
runs-on: ubuntu-latest
env:
DATABASE_URL: postgres://postgres:postgres@localhost/fang
CARGO_TERM_COLOR: always

strategy:
matrix:
toolchain:
- stable

services:
# Label used to access the service container
postgres:
# Docker Hub image
image: postgres
# Provide the password for postgres
env:
POSTGRES_PASSWORD: postgres
POSTGRES_USER: postgres
# Set health checks to wait until postgres has started

ports:
- 5432:5432

options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5

- name: Change working dir
working-directory: ./../..
run: pwd
steps:
- uses: actions/checkout@v4

- name: Setup Rust
run: rustup update ${{ matrix.toolchain }} && rustup default ${{ matrix.toolchain }}

- name: Install diesel-cli
run: cargo install diesel_cli --no-default-features --features postgres

- name: Setup Postgres db
working-directory: ./fang/postgres_migrations
run: diesel setup --database-url "postgres://postgres:postgres@localhost/fang"

- name: Run tests
uses: actions-rs/cargo@v1
with:
command: test
args: --verbose --all-features
run: cargo test "asynk::async_queue::postgres" --verbose --features asynk-postgres --color always -- --nocapture

- name: Run worker tests
run: cargo test "asynk::async_worker::async_worker_tests" --verbose --features asynk-postgres --color always -- --nocapture

test_sqlite:
name: Test sqlite
runs-on: ubuntu-latest
env:
CARGO_TERM_COLOR: always

strategy:
matrix:
toolchain:
- stable

- name: Run dirty tests
uses: actions-rs/cargo@v1
with:
command: test
args: --verbose --all-features -- --ignored
steps:
- uses: actions/checkout@v4

- name: Setup Rust
run: rustup update ${{ matrix.toolchain }} && rustup default ${{ matrix.toolchain }}

- name: Install sqlite3
run: |
sudo apt install -y sqlite3
sqlite3 fang.db "VACUUM;"
mkdir tests_sqlite

- name: Install diesel-cli
run: cargo install diesel_cli --no-default-features --features sqlite

- name: Setup Sqlite db
pxp9 marked this conversation as resolved.
Show resolved Hide resolved
working-directory: ./fang/sqlite_migrations
run: diesel setup --database-url "sqlite3://../../../fang.db"

- name: Run tests
run: cargo test "asynk::async_queue::sqlite" --verbose --features asynk-sqlite -- --nocapture

release:
name: Release x86_64-unknown-linux-gnu
runs-on: ubuntu-latest
needs: test
env:
CARGO_TERM_COLOR: always

steps:
- uses: actions/checkout@v3
strategy:
matrix:
toolchain:
- stable

steps:
- uses: actions/checkout@v4
- name: Setup Rust
uses: actions-rs/toolchain@v1
with:
override: true
profile: minimal
target: x86_64-unknown-linux-gnu
toolchain: stable
run: rustup update ${{ matrix.toolchain }} && rustup default ${{ matrix.toolchain }}

- name: Build release
uses: actions-rs/cargo@v1
with:
command: build
args: --release --verbose --all-features --target x86_64-unknown-linux-gnu
run: cargo build --release --verbose --all-features --target x86_64-unknown-linux-gnu

test_mysql:
name: Test mysql
runs-on: ubuntu-latest
env:
DATABASE_URL: mysql://root:mysql@localhost/fang
CARGO_TERM_COLOR: always

strategy:
matrix:
toolchain:
- stable

services:
# Label used to access the service container
mysql:
# Docker Hub image
image: mysql:8.1
# Provide the password for postgres
env:
MYSQL_ROOT_PASSWORD: mysql
MYSQL_DATABASE: fang
# here we should check if mysql is ready, but this does not work
options: >-
--health-cmd "mysqladmin ping -h localhost -u root -pmysql"
--health-interval 10s
--health-timeout 5s
--health-retries 5

ports:
- 3306:3306

steps:
- uses: actions/checkout@v4

- name: Setup Rust
run: rustup update ${{ matrix.toolchain }} && rustup default ${{ matrix.toolchain }}

- name: Install diesel-cli
run: cargo install diesel_cli --no-default-features --features mysql

- name: Setup MySQL db
working-directory: ./fang/mysql_migrations
run: diesel setup --database-url "mysql://root:[email protected]/fang"

- name: Run tests
run: cargo test "asynk::async_queue::mysql" --verbose --features asynk-mysql --color always -- --nocapture
19 changes: 9 additions & 10 deletions .github/workflows/style.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,24 +3,23 @@ name: Rust Code Formatting
on:
push:
pull_request:
types: [opened, reopened]

jobs:
rustfmt:
name: Rustfmt
runs-on: ubuntu-latest
env:
CARGO_TERM_COLOR: always
strategy:
matrix:
toolchain:
- stable
steps:
- uses: actions/checkout@v3

- name: Setup Rust
uses: actions-rs/toolchain@v1
with:
toolchain: stable
override: true
profile: minimal
components: rustfmt
run: rustup update ${{ matrix.toolchain }} && rustup default ${{ matrix.toolchain }}

- name: Check format
uses: actions-rs/cargo@v1
with:
command: fmt
args: -- --check --verbose
run: cargo fmt -- --check --verbose
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@ Cargo.lock
docs/content/docs/CHANGELOG.md
docs/content/docs/README.md
fang.db
tests_sqlite/
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ db_mysql:
db_sqlite:
@echo -e $(BOLD)Setting up SQLite database...$(END_BOLD)
sqlite3 "$(SQLITE_FILE)" "VACUUM;"
mkdir -p "$(SQLITE_TESTS_DIR)"
$(MAKE) diesel_sqlite

diesel: $(DIESEL_TARGETS)
Expand Down
2 changes: 1 addition & 1 deletion fang-derive-error/example/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ pub enum MyAwesomeError {
MyVariantErrorTwo(u32),
}
#[cfg(test)]
mod tests {
mod fang_derive_error_tests {
use crate::MyAwesomeError;
use fang::FangError;

Expand Down
Loading
Loading