← Index · ← File index · ← Prev · Next → · 📄 source

#882: ExtractPFTradingPartners.kt

projectforge-business/src/main/kotlin/org/projectforge/business/dvelop/ExtractPFTradingPartners.kt

Type: Spring @Service — Data extraction engine

Purpose: Extracts trading partners (vendors, customers) from ProjectForge's financial database tables and converts them into TradingPartner objects suitable for initial D-velop import.

Source path: projectforge-business/src/main/kotlin/org/projectforge/business/dvelop/ExtractPFTradingPartners.kt

385 lines · 319 code · 38 comments · 28 blank

Acts as the bridge between ProjectForge's legacy financial data (incoming invoices, outgoing invoices, and customer master data) and D-velop's trading partner model. It scans 5 years of invoice data, deduplicates entities across accounts, handles the "divers" (miscellaneous) account pattern, parses free-text billing addresses into structured fields, and classifies entities as VENDOR, CUSTOMER, or PARTNER. Also provides Excel export functionality for reconciliation.

Architecture

Inner Class: Context

A mutable accumulator that collects extracted trading partners during the multi-phase extraction process:

MemberDescription
vendorsMutableList<TradingPartner> — Creditor/vendor partners extracted from incoming invoices
customersMutableList<TradingPartner> — Customer partners extracted from outgoing invoices + customer master
sequenceAuto-incrementing counter starting at 7,000,000 for generating unique partner numbers
allPartnersCombined, deduplicated, sorted list of all partners (vendors + customers)
nextValReturns and increments the sequence counter
getVendorByDatevKonto(konto)Looks up a vendor by matching the konto number against importCode
cleanUp()Clears the temporary importCode field if it matches the partner number (no longer needed post-migration)

Extraction Phases

Phase 1: extractTradingVendors(invoices)

Processes EingangsrechnungDO (incoming/creditor invoices):

Phase 2: extractTradingCustomersInvoices(invoices, context)

Processes RechnungDO (outgoing/customer invoices):

Phase 3: extractTradingCustomers(customers, context)

Processes KundeDO (customer master data) for any customers not already captured via invoices. Calls context.cleanUp() at the end.

Core Deduplication Logic: handleKunde()

The most complex method in this service. For each KundeDO:

  1. Checks if a customer with the same number already exists → skips creation, optionally filling in datevKonto
  2. Checks if the konto matches an existing vendor → upgrades vendor to PARTNER, reassigns the customer number as the primary number
  3. Otherwise creates a new customer; active customers are those with ACQUISITION or ACTIVE status (or null)

Address Parsing: checkBillAddress()

Parses free-text billing addresses from invoice customerAddress strings using a bottom-up heuristic:

  1. Splits the address into non-blank lines
  2. Filters out leading lines matching known filler words ("Herr", "Frau", "Firma", "An", "Portal", "Klinikdienst")
  3. Last line: country or zip+city (tries extractZipCodeCity())
  4. Second-to-last: street line
  5. Any remaining lines prepended to billToAddressAdditional

extractZipCodeCity(line)

Uses a regex pattern ^[0-9 ]{5,10} to extract a German-style 5+ digit postal code. Strips "D-" and "D " country prefixes. Returns Pair<String, String> of (zipCode, city) or null if no match.

Helper Methods

MethodDescription
appendRemarks(partner, remarks)Appends text to partner.remarks with newline separator
prependRemarks(partner, konto)Prepends DATEV-Konto info (number + description) to remarks
checkIfGiven(vararg str)Returns true if any string is non-blank
lineEquals(line, vararg strs)Case-insensitive trimmed comparison
createVendor(number, kreditor, konto)Factory for vendor TradingPartners
createCustomer(...)Factory for customer TradingPartners

Excel Export: extractTradingPartnersAsExcel()

Creates a two-sheet Excel workbook using ExcelWorkbook:

Columns include number, datevKonto, importCode, type, company, shortName, remarks, active, and all billing address fields. Enum values (contactType, type, active) are resolved to their value string representation.

Injected Dependencies

DaoPurpose
EingangsrechnungDaoQuery incoming/creditor invoices
KundeDaoQuery customer master data
RechnungDaoQuery outgoing/customer invoices

Git History

868d6abb7 2025 -> 2026
78a38ca8f ExcelUtils.createFont and used by all font creating excel routines.
63081666f Source file headers: 2024-> 2025.
1b50060c3 BaseDao: renamed: get -> find, save -> insert, getList -> select, load -> select
87aaf6a5a Migration stuff in progress... BaseDao refactored (not yet finished) internal* methods renamed.
e33c8b9c2 Migration stuff in progress...
b6092df09 Copyright 2023 -> 2024
b710c3267 WIP D.velop
3ef41086d WIP D.velop
be750e77e WIP D.velop
5145a8199 WIP D.velop
9ebb99c88 WIP D.velop
cb2915283 WIP D.velop
b46787d5f WIP D-velop
2fe7d6f33 WIP D-velop
975af1cd6 WIP D-velop
1dee5f23f WIP D-velop
3ec5c7354 WIP D-velop
cb6d633f9 WIP D-velop
eb005b878 WIP D-velop
← Index · ← File index · ← Prev · Next → · 📄 source

#882: ExtractPFTradingPartners.kt

Type: Spring @Service — Data extraction engine

Purpose: Extracts trading partners (vendors, customers) from ProjectForge's financial database tables and converts them into TradingPartner objects suitable for initial D-velop import.

Source path: projectforge-business/src/main/kotlin/org/projectforge/business/dvelop/ExtractPFTradingPartners.kt

Acts as the bridge between ProjectForge's legacy financial data (incoming invoices, outgoing invoices, and customer master data) and D-velop's trading partner model. It scans 5 years of invoice data, deduplicates entities across accounts, handles the "divers" (miscellaneous) account pattern, parses free-text billing addresses into structured fields, and classifies entities as VENDOR, CUSTOMER, or PARTNER. Also provides Excel export functionality for reconciliation.

Architecture

Inner Class: Context

A mutable accumulator that collects extracted trading partners during the multi-phase extraction process:

MemberDescription
vendorsMutableList<TradingPartner> — Creditor/vendor partners extracted from incoming invoices
customersMutableList<TradingPartner> — Customer partners extracted from outgoing invoices + customer master
sequenceAuto-incrementing counter starting at 7,000,000 for generating unique partner numbers
allPartnersCombined, deduplicated, sorted list of all partners (vendors + customers)
nextValReturns and increments the sequence counter
getVendorByDatevKonto(konto)Looks up a vendor by matching the konto number against importCode
cleanUp()Clears the temporary importCode field if it matches the partner number (no longer needed post-migration)

Extraction Phases

Phase 1: extractTradingVendors(invoices)

Processes EingangsrechnungDO (incoming/creditor invoices):

Phase 2: extractTradingCustomersInvoices(invoices, context)

Processes RechnungDO (outgoing/customer invoices):

Phase 3: extractTradingCustomers(customers, context)

Processes KundeDO (customer master data) for any customers not already captured via invoices. Calls context.cleanUp() at the end.

Core Deduplication Logic: handleKunde()

The most complex method in this service. For each KundeDO:

  1. Checks if a customer with the same number already exists → skips creation, optionally filling in datevKonto
  2. Checks if the konto matches an existing vendor → upgrades vendor to PARTNER, reassigns the customer number as the primary number
  3. Otherwise creates a new customer; active customers are those with ACQUISITION or ACTIVE status (or null)

Address Parsing: checkBillAddress()

Parses free-text billing addresses from invoice customerAddress strings using a bottom-up heuristic:

  1. Splits the address into non-blank lines
  2. Filters out leading lines matching known filler words ("Herr", "Frau", "Firma", "An", "Portal", "Klinikdienst")
  3. Last line: country or zip+city (tries extractZipCodeCity())
  4. Second-to-last: street line
  5. Any remaining lines prepended to billToAddressAdditional

extractZipCodeCity(line)

Uses a regex pattern ^[0-9 ]{5,10} to extract a German-style 5+ digit postal code. Strips "D-" and "D " country prefixes. Returns Pair<String, String> of (zipCode, city) or null if no match.

Helper Methods

MethodDescription
appendRemarks(partner, remarks)Appends text to partner.remarks with newline separator
prependRemarks(partner, konto)Prepends DATEV-Konto info (number + description) to remarks
checkIfGiven(vararg str)Returns true if any string is non-blank
lineEquals(line, vararg strs)Case-insensitive trimmed comparison
createVendor(number, kreditor, konto)Factory for vendor TradingPartners
createCustomer(...)Factory for customer TradingPartners

Excel Export: extractTradingPartnersAsExcel()

Creates a two-sheet Excel workbook using ExcelWorkbook:

Columns include number, datevKonto, importCode, type, company, shortName, remarks, active, and all billing address fields. Enum values (contactType, type, active) are resolved to their value string representation.

Injected Dependencies

DaoPurpose
EingangsrechnungDaoQuery incoming/creditor invoices
KundeDaoQuery customer master data
RechnungDaoQuery outgoing/customer invoices

Git History

868d6abb7 2025 -> 2026
78a38ca8f ExcelUtils.createFont and used by all font creating excel routines.
63081666f Source file headers: 2024-> 2025.
1b50060c3 BaseDao: renamed: get -> find, save -> insert, getList -> select, load -> select
87aaf6a5a Migration stuff in progress... BaseDao refactored (not yet finished) internal* methods renamed.
e33c8b9c2 Migration stuff in progress...
b6092df09 Copyright 2023 -> 2024
b710c3267 WIP D.velop
3ef41086d WIP D.velop
be750e77e WIP D.velop
5145a8199 WIP D.velop
9ebb99c88 WIP D.velop
cb2915283 WIP D.velop
b46787d5f WIP D-velop
2fe7d6f33 WIP D-velop
975af1cd6 WIP D-velop
1dee5f23f WIP D-velop
3ec5c7354 WIP D-velop
cb6d633f9 WIP D-velop
eb005b878 WIP D-velop