Performance Test of Text File Processing with esProc

This article aims to test esProc’s performance in processing text files using an example of data query and filtering and through the comparison with Java and Perl performing the same processing. The orders.txt file storing order records is used as the test data. It is imported in esProc as follows: ORDERID CLIENT SELLE...

2016-05-27 4221 0 1

Rules of Resource Release in esProc

During data computing in esProc, raw data, intermediate data, and the final resulting data consume certain system resources. That includes the memory consumption, database connection usage and the space occupation by the file objects. Too much resource consumption may cause memory overflow, database access failure and ...

2016-05-20 5432 4 2

Reference-based Thinking in esProc

1.Foreign key reference The foreign key is a common concept for various relational databases. For a relational table, you can define one or multiple fields as the foreign key, through which an association with the data in another table can be created. For example, in a game scoring report, often players are recorded wi...

2016-05-17 2326 0 1

Index Sequences in esProc

With databases, creating appropriate indexes for tables can greatly increase query efficiency. Similarly, you can create index sequences for record sequences or table sequences in esProc to enhance the efficiency of querying data repeatedly. For example, you need to query the food order file Order_Foods.txt repeatedly....

2016-05-09 2177 0 0

Round-off Error Control with esProc

Numerical computations in data reporting are often done with some degree of accuracy by rounding off numerical values to their nearest numbers, which is one type of the rounding operations. Yet this can result in data inconsistency. And in that case, the rounded numbers need to be adjusted to restore the consistency. T...

2016-04-29 2337 0 1

Oracle & esProc Performance Comparison Testing on Single Machine

1. Testing purposes Test esProc and Oracle on the same hardware to compare their single machine performances in carrying out a single big task and small concurrent tasks. 2. Testing contents and methods Data volume: Small data volume: Single fact table around 10G. To prevent the testing result from being affected by th...

2016-04-25 1827 0 1

Date and Time Functions in esProc

Date and time data has its own characteristics for analysis and handling. In this article let’s look at how to perform date and time handling in esProc. 1. Conversion to date and time data Usually the date and time data is entered or displayed as strings. With esProc, you can click Tool>Option to set the default for...

2016-04-21 2083 0 0

The New Java Way of Handling Structured Text Files

It’s common to handle structured text files in Java programming. But since Java lacks corresponding class library, code produced through hardcoding is complicated and difficult to maintain. For example, sales.txt is a tab-separated structured text file that needs sorting by SellerId. Here’s the code: Comparator<...

2016-04-18 1779 0 0

Using Sequence Numbers and Locating Data in esProc

An esProc set is an ordered set. Therefore, its members can be referenced by sequence numbers. A flexible exploitation of sequence numbers will give full play to esProc’s capability, handling a computation in a simpler and more efficient way. 1. Accessing members Therefore it is recommended that certain esProc function...

2016-04-15 1798 0 0

esProc Variable Scope

esProc uses various variables, including the cell variable, in all its operations. Each type of variable has its scope, which is the topic the article discusses. 1. esProc Parameters and Variables esProc allows defining the globally-used data as cellset constant, like the following sequence-type constant weekdays: The...

2016-04-13 1878 0 0

Set-based Thinking in esProc

esProc applies set theory in a uniquely deep, common way, compared with traditional programming languages. The sequence in esProc is in essence a type of set. It’s important to learn to think in sets all the time when working with esProc. 1.Sequences and sets in esProc The sequence, as well as integer and string, is on...

2016-04-11 1843 0 0

Comparison operations in esProc

In data analysis, comparison operations are employed to check whether a datum is greater/less or equal to another datum, as well as to perform operations including query, select, sort and group. Here we discuss the uses of comparison operations in esProc and solve possible related problems. 1.Comparing single values es...

2016-04-08 1687 0 0

Structured-Data Computing Layer for Java

Programmers can handle structured data using SQL statements when the data is stored in a single database. But they can’t use SQL statements directly when the data is held in a non-database such as the file, the NoSQL database, or the Json data stream, or when multiple databases are involved. As an alternative, programm...

2016-04-06 1911 0 1

Regular Expressions in esProc

esProc provides regex() function to match a string or a string field value in a record sequence using a pattern defined by the regular expression. The aim is to analyze and examine the string in order to find the pattern in it and replace it. Here we look at the uses for regular expression in esProc. 1. The basics A re...

2016-04-01 1630 0 0

Examples of Handling Structured Text Files with esProc

Structured text files are a common file format, and the tasks of handling them are common too. One approach that comes to mind easily is to import such a file into the database for manipulation. But this is time-consuming and causes heavy database usage. In some cases, there is even not a database available. It’s conve...

2016-03-28 1621 0 0