Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FINERACT-2079: Rectify query for cashier transactions #3881

Merged
merged 1 commit into from
Oct 31, 2024

Conversation

wkigenyi
Copy link
Contributor

@wkigenyi wkigenyi commented May 5, 2024

Description

Describe the changes made and why they were made.

Ignore if these details are present on the associated Apache Fineract JIRA ticket.

Checklist

Please make sure these boxes are checked before submitting your pull request - thanks!

  • Write the commit message as per https://github.com/apache/fineract/#pull-requests

  • Acknowledge that we will not review PRs that are not passing the build ("green") - it is your responsibility to get a proposed PR to pass the build, not primarily the project's maintainers.

  • Create/update unit or integration tests for verifying the changes made.

  • Follow coding conventions at https://cwiki.apache.org/confluence/display/FINERACT/Coding+Conventions.

  • Add required Swagger annotation and update API documentation at fineract-provider/src/main/resources/static/legacy-docs/apiLive.htm with details of any API changes

  • Submission is not a "code dump". (Large changes can be made "in repository" via a branch. Ask on the developer mailing list for guidance, if required.)

FYI our guidelines for code reviews are at https://cwiki.apache.org/confluence/display/FINERACT/Code+Review+Guide.

@wkigenyi wkigenyi changed the title FINERACT-2079 Rectify query for cashier transactions FINERACT-2079: Rectify query for cashier transactions May 5, 2024
@adamsaghy
Copy link
Contributor

@wkigenyi Can you please write an integration test which calls the cashier transactions API and calls the cashier transactions with summary API? This way we can be sure, your fix is working and avoid regression in the future?

@wkigenyi
Copy link
Contributor Author

@adamsaghy this commit basically replaces the wrong table column in the sql query with the right one. Please guide me on the best way to write this this test.

@adamsaghy
Copy link
Contributor

@adamsaghy this commit basically replaces the wrong table column in the sql query with the right one. Please guide me on the best way to write this this test.

First of all thank you for the fix. Like I said in my earlier comment you should write an integration test which call the mentioned APIs and they are not failing (response is HTTP 200). This way we can make sure in the future if the query or the data model got changed we will know about it.

You may find plenty example in the "integration-test" directory.

Please let me know if you are facing any trouble!

@wkigenyi
Copy link
Contributor Author

Thanks Adam, I do understand the importance of the test. I will look into the mentioned directory for examples.

@adamsaghy
Copy link
Contributor

Thanks Adam, I do understand the importance of the test. I will look into the mentioned directory for examples.

Hi

I believe we dont really have too many test cases for cashier transactions, so i will try to give you some help here:

  • I would create a new Helper class which extends the IntegrationTest class which contains helper methods to call the "fineract-client" and via that we can call cashier transactions via type-safe way:
    Example:
    return ok(fineract().tellers.getTransactionsForCashier(tellerId, cashierId, currencyCode, ...))

You can see examples in the LoanTransactionHelper class, example in line 738

@wkigenyi
Copy link
Contributor Author

Thanks Alot Adam. I will at this in next few days. I have a commitment away from my machine.

@wkigenyi
Copy link
Contributor Author

Good Morning @adamsaghy. I tried to follow the example in LoanTransactionHelper, how to I run that test individually to see its results (either in intellij or gradlew)?

@bravosty
Copy link

could you help me on how to locate this directory, I am using mifos 23.12 and I have failed to locate the suggested file inorder to make the changes (appuser_id)

@wkigenyi
Copy link
Contributor Author

@adamsaghy please have a look at this.

@bravosty
Copy link

hey @wkigenyi , I am using the mifos 23 not docker version, the one downloaded from sourceforge.net/projects/mifos/
it runs in tomcat, is there any way of fixing that error from this kind of installation?
I noticed that the fixes apply to the cloned docker mifos version.

@wkigenyi
Copy link
Contributor Author

@bravosty you can locate the file fineract-provider/src/main/java/org/apache/fineract/organisation/teller/service/TellerManagementReadPlatformServiceImpl.java and make the change made in this commit.

@bravosty
Copy link

bravosty commented May 20, 2024 via email

@wkigenyi
Copy link
Contributor Author

@bravosty this has to be done in the source code. https://github.com/apache/fineract

@bravosty
Copy link

bravosty commented May 20, 2024 via email

@wkigenyi
Copy link
Contributor Author

@bravosty you would have to build your own jar/war. The alternative is to wait for for this PR to be merged and Fineract 1.10 released.

@bravosty
Copy link

bravosty commented May 20, 2024 via email

Copy link
Contributor

@adamsaghy adamsaghy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@adamsaghy adamsaghy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • What went wrong:
    You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
    Execution failed for task ':rat'.

A failure occurred while executing org.nosphere.apache.rat.RatWork
Apache Rat audit failure - 2 unapproved licenses
See file:///home/runner/work/fineract/fineract/build/reports/rat/index.html

Please add the proper licence header to the newly created files!
(You can copy from any existing file)

@wkigenyi
Copy link
Contributor Author

@adamsaghy license added

@adamsaghy
Copy link
Contributor

adamsaghy commented May 21, 2024

@wkigenyi Please kindly check the failing test case.

If you want to test locally, you can run the fineract backend and then configure your intelliJ to run the test case with IntelliJ Runner:
IntelliJ Settings -> Build, Execution, Deployment -> Build Tools -> Gradle -> Run tests using: Intellij IDEA

@wkigenyi
Copy link
Contributor Author

@adamsaghy I tried to run integration-tests with gradlew :integration-test:test and I was getting this error
The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.

@wkigenyi wkigenyi force-pushed the 2079 branch 2 times, most recently from cf27dec to 463b5dc Compare May 23, 2024 01:54
@wkigenyi
Copy link
Contributor Author

@adamsaghy I improved the test

@adamsaghy
Copy link
Contributor

@adamsaghy why are the last two checks here taking more than 24 hours ? 1.Fineract Build & Test - PostgreSQL / build 2.Fineract Docker build - PostgreSQL / build

Seems they stucked. I have restarted. I should be finished under 50 mins

@wkigenyi
Copy link
Contributor Author

@adamsaghy I have 1/8 failing
org.apache.fineract.client.util.CallFailedRuntimeException: HTTP failed: Request{method=GET, url=https://localhost:8443/fineract-provider/api/v1/tellers/1/cashiers/1/transactions?currencyCode=UGX&offset=0&limit=0, tags={class retrofit2.Invocation=org.apache.fineract.client.services.TellerCashManagementApi.getTransactionsForCashier() [1, 1, UGX, 0, 0, null, null]}}; Response{protocol=http/1.1, code=500, message=, url=https://localhost:8443/fineract-provider/api/v1/tellers/1/cashiers/1/transactions?currencyCode=UGX&offset=0&limit=0}; errorBody: <!doctype html><title>HTTP Status 500 – Internal Server Error</title><style type="text/css">body {font-family:Tahoma,
at app//org.apache.fineract.client.util.Calls.okR(Calls.java:65).
Could use some pointers here.

@adamsaghy
Copy link
Contributor

@wkigenyi Seems the cashier transaction API does not working with postgres database. Can you please take a look on it? It might requires different SQL for postgres database. You can find database dependent solution in the DatabaseSpecificSQLGenerator.

From the server logs it looks this exception:

org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar [select * from (select  txn.id as txn_id, txn.cashier_id as cashier_id,  txn.txn_type as txn_type,  txn.txn_amount as txn_amount, txn.txn_date as txn_date, txn.txn_note as txn_note,  txn.entity_type as entity_type, txn.entity_id as entity_id, txn.created_date as created_date,  o.id as office_id, o.name as office_name, t.id as teller_id, t.name as teller_name, s.display_name as cashier_name  from m_cashier_transactions txn  left join m_cashiers c on c.id = txn.cashier_id  left join m_tellers t on t.id = c.teller_id  left join m_office o on o.id = t.office_id  left join m_staff s on s.id = c.staff_id  where txn.cashier_id = ? and txn.currency_code = ? and o.hierarchy like ? AND ((case when c.full_day then Date(txn.created_date) between c.start_date AND c.end_date else ( Date(txn.created_date) between c.start_date AND c.end_date ) and ( TIME(txn.created_date) between TIME(c.start_time) AND TIME(c.end_time)) end) or txn.txn_type = 101))  cashier_txns  union (select  sav_txn.id as txn_id, null as cashier_id,  case      when renum.enum_value in ('deposit','withdrawal fee', 'Pay Charge', 'Annual Fee')          then 103      when renum.enum_value in ('withdrawal', 'Waive Charge', 'Interest Posting', 'Overdraft Interest', '')          then 104      else          105  end as txn_type,  sav_txn.amount as txn_amount, sav_txn.transaction_date as txn_date,  concat (renum.enum_value, ', Sav:', sav.id, '-', sav.account_no, ',Client:', cl.id, '-',cl.display_name) as txn_note,  'savings' as entity_type, sav.id as entity_id, sav_txn.created_date as created_date,  o.id as office_id, o.name as office_name, null as teller_id, null as teller_name, staff.display_name as cashier_name  from m_savings_account_transaction sav_txn  left join r_enum_value renum on sav_txn.transaction_type_enum = renum.enum_id and renum.enum_name = 'savings_transaction_type_enum'  left join m_savings_account sav on sav_txn.savings_account_id = sav.id  left join m_client cl on sav.client_id = cl.id  left join m_office o on cl.office_id = o.id  left join m_appuser user on sav_txn.created_by = user.id  left join m_staff staff on user.staff_id = staff.id  left join m_cashiers c on c.staff_id = staff.id  left join m_payment_detail payDetails on payDetails.id = sav_txn.payment_detail_id  left join m_payment_type payType on payType.id = payDetails.payment_type_id  left join m_account_transfer_transaction acnttrans  on (acnttrans.from_savings_transaction_id = sav_txn.id  or acnttrans.to_savings_transaction_id = sav_txn.id)  where sav_txn.is_reversed = false and c.id = ? and sav.currency_code = ? and o.hierarchy like ? and  sav_txn.transaction_date between c.start_date and date_add(c.end_date, interval 1 day)  and renum.enum_value in ('deposit','withdrawal fee', 'Pay Charge', 'withdrawal', 'Annual Fee', 'Waive Charge', 'Interest Posting', 'Overdraft Interest')  and (sav_txn.payment_detail_id IS NULL OR payType.is_cash_payment = true)  AND acnttrans.id IS NULL )  union (select  loan_txn.id as txn_id, c.id as cashier_id,  case      when renum.enum_value in ('REPAYMENT_AT_DISBURSEMENT','REPAYMENT', 'RECOVERY_REPAYMENT', 'CHARGE_PAYMENT')          then 103      when renum.enum_value in ('DISBURSEMENT', 'WAIVE_INTEREST', 'WRITEOFF', 'WAIVE_CHARGES')          then 104      else          105  end as cash_txn_type,  loan_txn.amount as txn_amount, loan_txn.transaction_date as txn_date,  concat (renum.enum_value, ', Loan:', loan.id, '-', loan.account_no, ',Client:', cl.id, '-',cl.display_name) as txn_note,  'loans' as entity_type, loan.id as entity_id, loan_txn.created_date as created_date,  o.id as office_id, o.name as office_name, null as teller_id, null as teller_name, staff.display_name as cashier_name  from m_loan_transaction loan_txn  left join r_enum_value renum on loan_txn.transaction_type_enum = renum.enum_id and renum.enum_name = 'loan_transaction_type_enum'  left join m_loan loan on loan_txn.loan_id = loan.id  left join m_client cl on loan.client_id = cl.id  left join m_office o on cl.office_id = o.id  left join m_appuser user on loan_txn.created_by = user.id  left join m_staff staff on user.staff_id = staff.id  left join m_cashiers c on c.staff_id = staff.id  left join m_payment_detail payDetails on payDetails.id = loan_txn.payment_detail_id  left join m_payment_type payType on payType.id = payDetails.payment_type_id  left join m_account_transfer_transaction acnttrans  on (acnttrans.from_loan_transaction_id = loan_txn.id  or acnttrans.to_loan_transaction_id = loan_txn.id)  where loan_txn.is_reversed = false and c.id = ? and loan.currency_code = ? and o.hierarchy like ? and  loan_txn.transaction_date between c.start_date and date_add(c.end_date, interval 1 day)  and renum.enum_value in ('REPAYMENT_AT_DISBURSEMENT','REPAYMENT', 'RECOVERY_REPAYMENT','DISBURSEMENT', 'CHARGE_PAYMENT', 'WAIVE_CHARGES', 'WAIVE_INTEREST', 'WRITEOFF')  and (loan_txn.payment_detail_id IS NULL OR payType.is_cash_payment = true)  AND acnttrans.id IS NULL )  union (select  cli_txn.id AS txn_id, c.id AS cashier_id,  case  when renum.enum_value in ('PAY_CHARGE')  then 103  when renum.enum_value in ('WAIVE_CHARGE')  then 104  else  105  end as cash_txn_type,  cli_txn.amount as txn_amount, cli_txn.transaction_date as txn_date,  concat (renum.enum_value, ', Client:', cl.id, '-', cl.account_no, ',Client:', cl.id, '-',cl.display_name) as txn_note,  'client' as entity_type, cl.id as entity_id, cli_txn.created_date as created_date,  o.id as office_id, o.name as office_name, null as teller_id, null as teller_name, staff.display_name as cashier_name  from m_client_transaction cli_txn  left join r_enum_value renum on cli_txn.transaction_type_enum = renum.enum_id AND renum.enum_name = 'client_transaction_type_enum'  left join m_client cl on cli_txn.client_id = cl.id  left join m_office o on cl.office_id = o.id  left join m_appuser user on cli_txn.created_by = user.id  left join m_staff staff on user.staff_id = staff.id  left join m_cashiers c on c.staff_id = staff.id  left join m_payment_detail payDetails on payDetails.id = cli_txn.payment_detail_id  left join m_payment_type payType on payType.id = payDetails.payment_type_id  where cli_txn.is_reversed = false and c.id = ? and cli_txn.currency_code = ? and o.hierarchy like ? and cli_txn.transaction_date  between c.start_date and date_add(c.end_date, interval 1 day)  and renum.enum_value in ('PAY_CHARGE', 'WAIVE_CHARGE')  and (cli_txn.payment_detail_id IS NULL OR payType.is_cash_payment = true) )  order by created_date ]
	at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:112)
	at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:107)
	at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:116)
	at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1548)
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:677)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:723)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:754)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:767)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:825)
	at org.apache.fineract.infrastructure.core.service.PaginationHelper.fetchPage(PaginationHelper.java:44)
	at org.apache.fineract.organisation.teller.service.TellerManagementReadPlatformServiceImpl.retrieveCashierTransactions(TellerManagementReadPlatformServiceImpl.java:504)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:351)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:713)
	at org.apache.fineract.organisation.teller.service.TellerManagementReadPlatformServiceImpl$$SpringCGLIB$$0.retrieveCashierTransactions(<generated>)
	at org.apache.fineract.organisation.teller.api.TellerApiResource.getTransactionsForCashier(TellerApiResource.java:324)

@galovics
Copy link
Contributor

@wkigenyi are you planning to fix the PostgreSQL version of the query?

@wkigenyi
Copy link
Contributor Author

Yes, though I would have preferred to write something that is dB independent. I am just a bit stuck but will see it through.

@galovics
Copy link
Contributor

galovics commented Sep 6, 2024

@wkigenyi I agree but it's a good strategy to first get it working and merged. Then as a step 2, you could try to change the query so it's DB independent.

@wkigenyi
Copy link
Contributor Author

wkigenyi commented Sep 6, 2024

Okay. Will find time for this and resolve it.

@galovics
Copy link
Contributor

@wkigenyi any update here?

@wkigenyi
Copy link
Contributor Author

wkigenyi commented Oct 4, 2024

@wkigenyi any update here?

I have been a bit swamped. I have time this weekend to work on it.

@galovics
Copy link
Contributor

galovics commented Oct 4, 2024

@wkigenyi superb, as soon as you're ready, I can review it.

@wkigenyi
Copy link
Contributor Author

wkigenyi commented Oct 4, 2024

@wkigenyi superb, as soon as you're ready, I can review it.

There are two ways to create a cashier

  1. Full Day
  2. Not full day (you have to specify the startTime and endTime)

The second option does not seem to be supported anymore because when you provide values for fields when creating the cashier you get an error that they are not supported, when you leave them out in the request without setting isFullDay to true you will get an error message saying startTime is required.
I have noticed that the web-app UI does not have provision for startTime and endTime.

Should I therefore assume that these fields are no longer used when creating the cashier and rewrite the query without considering them?

@wkigenyi
Copy link
Contributor Author

wkigenyi commented Oct 8, 2024

@wkigenyi superb, as soon as you're ready, I can review it.
I have made some improvements but I am not sure if I squashed this correctly.

@wkigenyi wkigenyi marked this pull request as draft October 8, 2024 14:57
@wkigenyi wkigenyi marked this pull request as ready for review October 8, 2024 19:38
@galovics
Copy link
Contributor

galovics commented Oct 9, 2024

@wkigenyi formatting is off, please run a ./gradlew spotlessApply on the code.

@wkigenyi
Copy link
Contributor Author

wkigenyi commented Oct 9, 2024

It fails with

Task :fineract-tax:spotlessMiscApply FAILED

fineract-provider/hs_err_pid4140.log Outdated Show resolved Hide resolved
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public final class CashierHelper {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mind not creating yet another helper class for the tests but rely on the fineract-client that generates the API automatically?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking into this

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public final class TellerHelper {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same, generated API classes?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking into this

sqlBuilder.append(" and txn.currency_code = ? ");
sqlBuilder.append(" and o.hierarchy like ? ) cashier_txns ");
// sqlBuilder.append(" and o.hierarchy like ? ");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's get rid of this commented line if it's not needed.

@@ -762,8 +737,8 @@ public String cashierTxnSummarySchema() {
sqlBuilder.append(" or acnttrans.to_savings_transaction_id = sav_txn.id) ");
sqlBuilder.append(" where sav_txn.is_reversed = false and c.id = ? ");
sqlBuilder.append(" and sav.currency_code = ? ");
sqlBuilder.append(" and o.hierarchy like ? ");
sqlBuilder.append(" and sav_txn.transaction_date between c.start_date and date_add(c.end_date, interval 1 day) ");
// sqlBuilder.append(" and o.hierarchy like ? ");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

@@ -801,8 +776,8 @@ public String cashierTxnSummarySchema() {
sqlBuilder.append(" or acnttrans.to_loan_transaction_id = loan_txn.id) ");
sqlBuilder.append(" where loan_txn.is_reversed = false and c.id = ? ");
sqlBuilder.append(" and loan.currency_code = ? ");
sqlBuilder.append(" and o.hierarchy like ? ");
sqlBuilder.append(" and loan_txn.transaction_date between c.start_date and date_add(c.end_date, interval 1 day) ");
// sqlBuilder.append(" and o.hierarchy like ? ");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

sqlBuilder.append(" left join m_cashiers c ON c.staff_id = staff.id ");
sqlBuilder.append(" left join m_payment_detail payDetails on payDetails.id = cli_txn.payment_detail_id ");
sqlBuilder.append(" left join m_payment_type payType on payType.id = payDetails.payment_type_id ");
sqlBuilder.append(" where cli_txn.is_reversed = false AND c.id = ? ");
sqlBuilder.append(" and cli_txn.currency_code = ? ");
sqlBuilder.append(" and o.hierarchy LIKE ? ");
sqlBuilder.append(" and cli_txn.transaction_date between c.start_date and date_add(c.end_date, interval 1 day) ");
// sqlBuilder.append(" and o.hierarchy LIKE ? ");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All Resolved

Copy link
Contributor

@adamsaghy adamsaghy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@galovics galovics merged commit d202145 into apache:develop Oct 31, 2024
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants