asfernandes

Como se tornar (ou não) um "Firebird Developer"

2011-10-22T01:15:00.001-02:00

Recentemente eu divulguei em meu twitter que estava fazendo sete anos do meu primeiro commit no projeto Firebird. Daí citaram isso no FirebirdNews e em newsletter da Firebase.

Estou dizendo isso porque apareceram algumas pessoas com aquele tipo comum de questionamento "Como entrar para o time Firebird?". O problema é que, como citado na newsletter da Firebase, desses sete anos pra cá nenhum desenvolvedor se juntou ao Firebird (core). Isso mesmo, com exceção de um ou outro patch específico (geralmente dos mantenedores das distros Linux), o Firebird é desenvolvido pelas mesmas poucas pessoas há mais de sete anos.

Sempre respondi esses e-mails, mas geralmente a "vontade" da pessoa já termina aí. E olha que sempre respondi com mais educação do que estou usando pra escrever esse post.

Então vou colocar o que realmente penso sobre isso. Em primeiro lugar, você não deve fazer essa pergunta "Como entrar para o time Firebird?". A primeira coisa é aprender a usar o Google e se surgir alguma dúvida que não tenha uma resposta, aí sim pergunte sobre ela.

É normal que você tenha dificuldade pra "se inserir" no projeto. Eu comecei a acompanhar o desenvolvimento do Firebird em dezembro de 2003 e meu primeiro commit foi em setembro de 2004, apesar de estar trabalhando ativamente desde junho ou julho. O Claudio Valderrama costumava dizer que eu era o único que tinha aprendido o código em algumas semanas, mas como podem ver foi alguns meses :), sem falar que até hoje eu não gosto nem de ver certos subsistemas (hello PAG, CCH, SDW...).

Outra coisa são suas habilidades. Não é normal que você não tenha no mínimo bons conhecimentos sobre C++, que é a linguagem usada no desenvolvimento do Firebird. Eu posso te afirmar com 98% de certeza que você não vai encontrar em um projeto open-source pessoas com vontade de te ajudar com coisas básicas.

Digamos que você já esteja pensando que não quer isso (como todos os anteriores), mas tenha alguma dúvida ainda, então continuo minhas sugestões para que você não consiga :). Não leia todos os posts das listas de discussões. Também não leia e tente entender todas as alterações feitas no código, que são enviadas pra lista firebird-checkins. Acho que foi aqui que falhei, pois desde aquele dezembro de 2003 não deixei de ler nenhuma das milhares de mensagens. E tenho certeza que todos os outros desenvolvedores ativos fizeram e fazem o mesmo.

Se você também se ofendeu com esse post (caso tenha sido um dos que fez aquela pergunta), é melhor não perder seu tempo e desistir logo. Quem acompanha as listas firebird-devel e firebird-architect (e a firebird-admins, mas essa é fechada), sabe do que estou falando. As vezes (pra não dizer, muitas vezes) as discussões partem pro lado pessoal, e isso é a pior parte de fazer parte do projeto. Isso não é uma exclusividade do Firebird. Comparando com o Linux, por exemplo, é só procurar alguma mensagem do Linus Torvals na lkml e ver como a coisa geralmente funciona.

Beautiful fonts in Ubuntu

2011-08-20T12:58:00.004-03:00

In the old days, fonts in Linux were really ugly by default. There are patents involved, so it was possible to make them beautiful recompiling freetype with some options.

Since some time, this is not necessary.

But if you're like me, you may note that some fonts are very blurry in some applications, and you can't change that in the appearance config.

To fix this, I put this .fonts.conf in my home directory.

Ubuntu Natty Narwhal in VMware

2011-05-01T14:05:00.005-03:00

Here are some tips to use Ubuntu 11.04 Natty Narwhal on VMware.

First upgrade the host vmware (player) to the latest (3.1.4) version. Although the release notes mention nothing on Natty, the vmware-tools from this version compiles ok on the Natty kernel.

You may have some luck with the open-vm-tools. I didn't had much. It didn't recognized the machine as a VM, but still worked. I was having problems with the video looking very ugly and was thinking it was a vmware-tools problem, but looks like not. Below is a workaround to run on the terminal which makes things beautiful again:

killall gnome-settings-daemon

gnome-settings-daemon &

Alguns links de artigos e vídeos que acho interessantes

2011-04-20T14:54:00.006-03:00

Segue alguns links de artigos e vídeos que acho interessantes:

Porque não se trabalha no trabalho - Jason Fried tem uma teoria radical sobre o trabalho, que diz que o escritório não é um bom lugar para fazê-lo.

A Arte do Começo - Vídeo com Guy Kawasaki dando uma palestra sobre como se começar um negócio.

Verdades não tão conhecidas sobre programação - Tradução de um artigo que fala sobre a diferença entre programadores bons e ruins.

Net Negative Producing Programmer - Fábio Akita fala sobre as diferenças entre os Codificadores (cowboys) e os Desenvolvedores.

Ócio Criativo x Cowboy Coders - Vídeo onde Daniel Cukier coloca a sua visão sobre a diferença entre Codificadores e Programadores, em alusão ao post (acima) do Fábio Akita.

The Future Java Developer - O JavaMan Bruno Souza fala sobre o que desenvolvedores podem fazer para serem melhores profissionais no futuro (presente). Não é específico de Java.

Novidades do Firebird 2.5 - Revista ClubeDelphi

2011-01-01T20:05:00.006-02:00

Saiu na edição 125 da revista ClubeDelphi o artigo que escrevi sobre as novidades do Firebird 2.5. Esta edição também tem outra matéria sobre Firebird, entitulada "Aplicações com Firebird Embedded – Parte 1".

Já tinha recusado convites similares, mas desta vez resolvi topar o desafio. Não gosto muito de escrever, sou bastante perfeccionista e português não é uma das linguagens que eu domino (rs). Essa é uma das razões porque eu prefiro escrever em inglês no blog. Não que meu inglês seja bom, mas eu me sinto mais a vontade pra escrever mal. :)

Não vi a revista ainda. Espero que gostem, deu bastante trabalho.

Introducing the boolean datatype

2010-12-18T10:36:00.002-02:00

The support for a BOOLEAN datatype was the third most voted feature in the Firebird tracker. The deal with it was about make it right and complete, having booleans expressions allowed where values are expected, and vice versa. Oracle did it wrong and they BOOLEAN is crap.

Yesterday I finished and committed this support for Firebird 3. You can use them like any other type. You can index it, aggregate by it and sort by it. If you have an index on a boolean field, you can also do indexed searches by its negated form (NOT field).

Three new literals are introduced: FALSE, TRUE and UNKNOWN (same as NULL).

Booleans are not implicitly convertible to any other datatype. But it's convertible to/from strings with CAST.

It's allowed to test booleans without compare with TRUE or FALSE. For example, "field1 OR field2" and "NOT field1" are valid expressions. It's also allowed to compare with others operators, including the new IS operator: "field1 IS FALSE".

Currently it has a terrible and bad hack. The new syntax would introduce a lot of parser conflicts related with trigger's INSERTING, UPDATING and DELETING expressions. These boolean expressions was non-reserved words and allowed to be used as column and variable names. The correct solution shall be to reserve these words and do not allow them to be used as names without double quotes.

As usual, this causes conflict opinions. I don't want to fight forever on this, so I make them work as triggers keywords in boolean expressions and as values in non-booleans expressions. This is much very compatible with previous versions, but is very confusing. For example, in "SELECT inserting, NOT inserting FROM test WHERE INSERTING and INSERTING IS TRUE" the INSERTING's are recognized as value, keyword, keyword and value.

I hope this clears people's minds and we have a better solution in the final version.

Problema com Java 1.6.0_22 e NF-e de SP

2010-12-14T19:18:00.006-02:00

Tivemos um problema, em ambiente interno de testes, com os webservices da Nota Fiscal Eletrônica do estado de São Paulo. O sistema estava sendo migrado de sistema operacional HP-UX (com Java distribuído pela HP) para um novo servidor com Red Hat Enterprise Linux (com Java da ~~Sun~~ Oracle).

Ao acessar os webservices a aplicação congelava ou dava erros de "read timed out". O primeiro passo foi verificar se o servidor estava acessando a internet normalmente. Através do elinks era possível acessar sites via http e https normalmente. O webservice retornava corretamente o erro de falta de certificado cliente.

Depois tentei fazer conexões pela aplicação adicionando "?WSDL" às URLs. Nem o ambiente de homologação nem o de produção da NF-e funcionavam. Outros endereços http funcionavam normalmente pela aplicação. Pra debugar o problema, adicionamos a opção -Djavax.net.debug=all. A requisição era enviada mas não vinha a resposta.

Resolvi fazer o teste com os webservices de contingência do SCAN e funcionou. Analisando os logs da conexão, resolvi procurar sobre o termo no_renegotiation que estava aparecendo quando acontecia o erro.

Esta página esclareceu o problema, que foi causado devido a uma alteração no Java para prevenir uma vulnerabilidade do protocolo TLS. Adicionamos as opções -Dsun.security.ssl.allowUnsafeRenegotiation=true e sun.security.ssl.allowLegacyHelloMessages=true até que o ambiente de SP seja atualizado.

Firebird 2.5 Introduces New Audit Features and Improved Scalability

2010-10-04T21:16:00.009-03:00

October 4, 2010

The Firebird Project today announces Firebird 2.5, the fifth and newest major release of its open source relational database management system. Enhancements in this new release are pitched at better serving the needs of businesses of any scale, from embedded device to enterprise resource planning solutions.

Philippe Makowski, President of the non-profit Firebird Foundation observes that the Firebird 2.5 release is a very important step in Firebird's 10th anniversary year, not just for the Firebird Project but for the whole open-source world, too.

"Today businesses all around the world are looking for mature, cost-effective solutions and Firebird 2.5 offers them a solution that is truly powerful and truly free", he noted.

The mix of high performance, small footprint, supreme scalability, silent and simple installation and royalty-free deployment make Firebird a highly attractive choice for all types of software developers and vendors. Firebird deployments are well established around the globe, serving data to hundreds of thousands of business systems with hundreds of users and databases exceeding 300GB.

Known Firebird installations number more than 2,000 each day, according to download statistics.

"Many of our customers have Firebird databases with sizes from 200GB to 400GB and they keep growing", said Stewart Spink, CTO of Watermark Software. "The performance improvements in Firebird 2.5 will ensure that their future demands will be satisfied completely."

Alexander Shaposhnikov, CIO of medical distributor Profitmed predicts that Firebird 2.5 will support the growth of Profitmed's business. "Currently we have a 65Gb database and 250 active users working 24x7. We intend to double both the size of the database and the number of users in two years and we have chosen Firebird 2.5 for this business-critical task."

Firebird binaries are distributed for a variety of system and hardware platforms: Windows, Linux, MacOS, Solaris, HP-UX and more. Firebird runs on on x86, x64, PowerPC, Sparc and other hardware platforms, providing an easy migration mechanism between these diverse platforms.

The V.2.5 release binaries ship for 32-bit and 64-bit Windows and Linux, with MacOSX x86 to follow. Other POSIX platform builds are likely to appear before the year is out, in response to demand.

New features of Firebird 2.5

New SuperClassic Architecture Firebird 2.5 introduces a new architecture, tagged "SuperClassic", to gain better leverage from multi-core and multi-CPU hardware environments and improve resource usage for systems with large numbers of users and huge databases.

Audit System audit tracing and user trace sessions via the Services API enable nearly real-time monitoring and analysis of everything going on in a database

Cross-database queries Firebird 2.5 opens the way for Firebird database instances to query one another and to exchange information

Enhanced user management User management becomes accessible and flexible through SQL requests submitted from user databases

Other features include autonomous transactions within PSQL modules (stored procedures, triggers, SQL blocks), support for regular expressions as arguments in SQL using the SIMILAR TO predicate, asynchronous cancellation of connections, enhancements of monitoring capabilities and much more.

Paul Beach, President and CEO of IBPhoenix, an organisation that offers professional services and contributes to Firebird development, considers the new security features and the impressive scaling capability of Firebird 2.5 will be welcomed by many existing Firebird users.

"In particular, it will help those in the finance and health industries to meet the growing requirements of their businesses", he notes, "especially in pushing the limits of performance."

David Wilder, CTO of Bas-X, welcomed the Firebird 2.5 release for its improved useability and flexibility. "It will enable our customers to turn over more transactions and run more concurrent users on the same hardware," he said.

Pricing and Availability

Firebird 2.5 is an open source database system that is available free of charge for any kind of usage, be it commercial, educational, non-profit or simply for private use. Both binary packages and the complete source code can be downloaded at no cost through the Firebird Project website for immediate installation. No registration or activation is required.

About the Firebird Foundation

The Firebird Foundation is a non-profit organization with the goal to support the development and growth of the Firebird relational database system. The Foundation was incorporated in 2002 and is currently supported by more than 300 active members, several of whom are also cash sponsors. It also accepts and manages donations of cash and resources from persons and organisations that are not regular members. The Foundation redistributes these funds as grants to some of the key workers in the Firebird Project. The Firebird Project has no other source of income.

Media Contact: Alexey Kovyazin * +7 910 402 94 34 * admin AT mindthebird DOT com

Firebird 2.5 launch

2010-09-30T20:53:00.003-03:00

Firebird 2.5 release date is set - it will be October 4th, 2010.

Please join Philippe Makowski, President of Firebird Foundation, and Dmitry Yemanov, lead Firebird developer, at the Launch Webinar devoted to the 5th major release of Firebird.

It will take place at October 4th, 2010, at 13:00 GMT (10:00 pelo horário de Brasília).

If you have any questions regarding Firebird 2.5, please feel free to contact Mind the Bird campaign at launch@mindthebird.com.

Firebird 10th anniversary. How I came to it.

2010-07-31T14:31:00.002-03:00

July 31th, 2010. Today Firebird completes its 10th anniversary. The Firebird website has a set of history documents about how that started. It’s something you must read if you’re interested in Firebird.

I’m sometimes asked about how I became one of its team members. So I’m going to tell something about this.

First things first. My first contact with InterBase was with the one present in the Delphi 3 CD. Searches says it was InterBase 4.2. I must say I didn’t like it too much. But my main problem at the time was with SQL. Not with its syntax, but with the idea of manipulating a database with text commands, and not via an API in a programming language. Fortunately, I was a child and grew up since then. :-)

The second important thing I remember about InterBase was about it becoming open source. This was something I liked, but just forgot soon later.

Then sometime later I came to the Info download site and it had Firebird with a text saying something like “Download it, while it’s free”. I downloaded it (the 1.0 version) and saw on its website that there was no plan to make it non-free. I started using and liked it. What most attracted me was it excellent transaction control mechanism.

I then started using the 1.5 RC versions and found a security bug on it. I tried to report on the SourceForge bug tracker without success. So I subscribed myself to firebird-devels mailing list and reported it there. That happened around in December 2003, and I never leaved the list since then. :-)

Lurking at firebird-devel was a challenge. I didn’t understand much English nor the technical talks, but I stayed there reading every message. I started to read the source code and understand the subsystems. I then started following firebird-checkins mailing list, where I could better understand how each feature/fix got implemented.

Around July 2004, I joined the Firebird PT_BR project to adapt for Firebird 1.5 the Brazilian collation done by Paulo Henrique Albanez for Firebird 1.0. I proposed different approach to definitively integrate it in Firebird and it was also rejected. So I got the work to do the “right” changes to the Firebird International Language (INTL) subsystem. In September 2004 I officially joined the project.

What’s more nice about being there is that I can work with very competent team, delivering a good software to millions of people. And just doing something I really like, designing features, programming in C++ and working with compiler (SQL / PSQL) implementation...

Long live Firebird!

Window functions (part3) - new functions

2010-02-19T23:46:00.001-02:00

In the first and the second part of window function series, I talked about the OVER clause, with/without partitions and with/without ordering. Till then, no new function had been introduced, so it has about the already existing aggregate functions.

Now, new window-only functions has been introduced. I'll separate them in two groups: ranking and navigational. Both set of functions can be used with/without partition/ordering, but them does not make much sense without ordering.

Ranking functions (DENSE_RANK, RANK and ROW_NUMBER): with these functions, one can create different type of incremental counters. Think about SUM(1) OVER (ORDER BY SALARY), these functions does this type of thing, but all of them in different ways. Lets see an example query, also comparing with the SUM behavior.

select
id,

salary,

   dense_rank() over (order by salary),
   rank() over (order by salary),
   row_number() over (order by salary),
   sum(1) over (order by salary)
  from employee

order by salary;

And the result set:

id	salary	dense_rank	rank	row_number	sum
3	8.00	1	1	1	1
4	9.00	2	2	2	2
1	10.00	3	3	3	4
5	10.00	3	3	4	4
2	12.00	4	5	5	5

As you see, the functions differs when repeated values are found in the order key (the salary of 10.00). With DENSE_RANK, no gaps are created and all repeated values receive the same counter. With RANK, the initial counter is used for all repeated values, but it make gaps so the next non-repeating value (the salary of 12.00) does not consider the previously repeated values as in different positions. With ROW_NUMBER, each line receives an incremental value. And SUM(1) is very like RANK, but the value is computed after all repeated values are summed, so the gap is before the repeated values.

Navigational functions (LAG and LEAD): with these functions, an expression could get the value of a previously (LAG) or a posterior (LEAD) row of the query. Follow the demonstrating query.

select

id,

salary,

lag(salary) over (order by salary),

lead(salary) over (order by salary)

from employee

order by salary;

And the result set:

id	salary	lag	lead
3	8.00	<null>	9.00
4	9.00	8.00	10.00
1	10.00	9.00	10.00
5	10.00	10.00	12.00
2	12.00	10.00	<null>

The functions have a second argument, that is implicitly 1, which means the number of previously/posterior rows will be queried. If this row does not exist, <null> is returned.

And of course, you can use these functions as expressions and compute, for example, the difference of an employee salary from the one greater and/or lesser.

Firebird History in 9 minutes video

2010-02-13T23:28:00.004-02:00

This is a code_swarm of the firebird2 tree.

Window Functions (part2) - cumulative aggregates

2010-01-24T16:59:00.001-02:00

Continuing with the window functions support for Firebird 3, I have now added another OVER sub-clause: ORDER BY. The ORDER BY sub-clause can be used with or without partitions, and it makes aggregate functions return the partial aggregations as the records are being processed.

Considering our employee table has this data:

id	salary
1	10.00
2	12.00
3	8.00
4	9.00
5	10.00

And we run this query:

select
id, salary, sum(salary) over (order by salary) cum_salary
from employee

order by salary;

The result set produced will be:

id	salary	cum_salary
3	8.00	8.00
4	9.00	17.00
1	10.00	37.00
5	10.00	37.00
2	12.00	49.00

So cum_salary returns the partial/accumulated aggregation (of the SUM function). You may found strange the 37.00 repeated for the ids 1 and 5, but that is how it should work. The ORDER BY keys are grouped together and the aggregation is computed once (but summing the two 10.00).

You can use multiple windows with different orders, and ORDER BY parts like DESC, NULLS LAST, etc. Here is a weird example:

select
   id, salary,
   sum(salary) over (order by salary) cum_salary,
   sum(salary) over (order by salary desc) cum_salary_desc
  from employee
  order by salary;

The result set produced will be:

id	salary	cum_salary	cum_salary_desc
3	8.00	8.00	49.00
4	9.00	17.00	41.00
1	10.00	37.00	32.00
5	10.00	37.00	32.00
2	12.00	49.00	12.00

With a partition ORDER BY works the same way, but at each partition boundary the aggregation is reset.

All aggregation functions are usable with ORDER BY, except the LIST function. As LIST returns blob, it would be slow to return multiple blobs (one for each order group), trick to implement, and I don't see an usage scenario for it.

Window Functions

2010-01-19T00:29:00.001-02:00

By the SQL specification, window functions are a kind of aggregation, but which does not “filter” the result set of a query. The aggregated data is mixed with the query result set. That sort of functions are used with the OVER clause. Users of Oracle also knows window functions as analytical functions.

We have promissed very basic support for window functions in Firebird 3, which was the OVER () clause using the current aggregate functions. With the OVER () clause, one can mix with the query result set the aggregated data over the entire result set. Let me explain with an example.

We have a table EMPLOYEE with columns ID, NAME and SALARY, and want to show each employee with his respective salary and the percentage of his salary over the payroll. With a “normal” query, we would do:

select

id,

name,

salary,

salary / (select sum(salary) from employee) percentage

from employee;

We need to repeat ourselves and wait so much to see the results. We can also make it hopefully faster using a cross join, but still the whole employee table will need to be read more than one time. Change the table by a complex view or add various “windows” and we'll have a performance problem for sure.

The same query could be specified in much more elegant and faster way using a window function. Here is it how:

select

id,

name,

salary,

salary / sum(salary) over () percentage

from employee;

Here, sum(salary) over () is computed with the sum of all SALARY from the query (the employee table). This is what the OVER () clause does.

But the OVER clause is not just that, and now I've added another of its subclauses to Firebird. It's the PARTITION subclause.

A partition is a way to make the OVER aggregation based on a GROUP. Its syntax is:

<window function>([<expr>]) OVER (PARTITION BY <expr> [, <expr> ...])

So now since the aggregation is done over a group, it could produce more than one row. So the result set generated by a partition is joined with the main query using the same expression list of the partition.

For a example, lets add to our employee table a ROLE column. Now we want the same information, but instead of see the percentage of the employee salary over all employees, we want to see that value based on the employees occuping the same role. Here is how that query could be written:

select

id,

name,

role,

salary,

salary / sum(salary) over (partition by role)

from employee;

With the current implementation, the query (without the window functions) is executed one time and cached. Extra aggregation is done on this cached data. Partitions are joined by the new hash join algorithmn, and the main window (OVER without a partition) is joined through a cross join with the main query.

There is much more about window functions, but this subset seems to cover some major use cases of them.

Integração da JVM ao Firebird

2009-12-10T22:11:00.001-02:00

A um tempo atrás "descobri" a integração do Oracle com Java no lado servidor, e logo pensei: O Firebird precisa disso também.

Sabia da iniciativa anterior de outros desenvolvedores fazerem o mesmo, e a idéia foi fazer o projeto vingar. Por diversos motivos isso pareceu impossível, então resolvi iniciar uma implementação alternativa (o FB/Java), bastante diferente internamente e com mais recursos (inclusive que o Oracle).

Usei este projeto como trabalho de conclusão de curso (TCC) de Ciência da Computação, e estou disponibilizando o mesmo para download.

Inicio explicando os conceitos e funcionalidades do Firebird e do Java relevantes ao tema. Depois, explico o funcionamento da integração entre as duas tecnologias.

Ubuntu 9.10 review

2009-11-02T14:44:00.001-02:00

I've upgraded Ubuntu 9.04 to 9.10, and here is what I found about it.

At upgrade start, it asked about keyboard. Surprisingly, it has nothing about Brazilian keyboards and I choose to maintain "kernel keymaps" without know what exactly that is and why that keyboard popup appeared.

During the upgrade, flash_plugin and package_hook updates crashed. Nvidia driver, as always failed too. So after restarting in safe mode, I changed nvidia driver to nv to be able to enter in the graphical mode.

Then, to fix the system:
   sudo aptitude safe-upgrade
   sudo aptitude safe-upgrade
   sudo aptitude dist-upgrade

And reverted to nvidia driver.

They claim to have new beautiful and faster boot. Lies. The new boot screen is horrible, specially when running fsck. And appears to be slower than in 9.04.

The "fast user switch" gnome applet do not exist as before. It can't show your IM status anymore, nor can be compact as before. I changed it by the shutdown applet to have the same functionality that I used to have.

As with Ubuntu 9.04, Firefox 3.5 continues slow as the 9.04 Shiretoko in my system. And the flash plugin continue crashing. I use the AMD64 version, which requires wrapper to 32 bit flash, because Adobe didn't released the final AMD64 version yet.

Resuming, you decide. I want an updated system anyway. :-)

The buggy const

2009-08-09T12:45:00.001-03:00

In C/C++, there are constants and pointer to constants. I'm talking here about constants (i.e., const int var1; int* const var2) and not pointer to constants (i.e., const int* var3).

In 90% of cases that I see, usage of constants for local "variables" and parameters is "wrong". People seems to use it for a variable that is not changing, while, in my opinion, it should rather be used for variables that should never change.

Anyway, I always thought that constant parameters were not part of function signatures, because a const parameter has nothing to do with the caller, but is routine implementation detail. Indeed, this code compiles, links and run with G++ and MSVC++ and is also validated by the Comeau online validator:

#include <stdio.h>

class Test
{
public:
   void m1(const int n);
   void m2(int n);
};

int main()
{
   Test t;
   t.m1(10);
   t.m2(10);
   return 0;
}

void Test::m1(int n)
{
   printf("%d\n", n);
}

void Test::m2(const int n)
{
   printf("%d\n", n);
}

But not all compilers agree. It has been sent some patches fixing these inconsistencies to compile Firebird with Solaris Sun Studio compiler. This usage seems to cause problems because the compiler treat the declaration and implementation as different signatures.

In Java, final could be used for the same thing. Let see what it think about:

interface Int
{
    public void m1(final int n);
    public void m2(int n);
}

class Test implements Int
{
    public void m1(int n)
    {
    }

    public void m2(final int n)
    {
    }
}

Java also accepts this code.

DDL execution architecture in Firebird - why we need to move on

2009-08-08T19:17:00.001-03:00

Here I'm going to explain how Firebird DDL commands works in the architecture, why it stops innovation and how it is supposed to work in Firebird 3.0.

DDL in FB works more or less like DML, so first a briefly explanation of how DML works. When a DML command is prepared, it starts in the parser constructing a tree of nodes. That nodes are all a single pointer type, used for all node types and others usages (like storing constants). A node have a list of child nodes.

After parsing, it enters in the semantics phase, where things are verified and adjusted with execution in mind. After that, it enters in the generation phase, that outputs BLR bytes and stores them in the statement.

The DSQL API is a layer around the engine API. The engine API knows only how to execute BLR. Hence the DSQL API does passes the stored BLR to that lower level functions (engine API).

For engine API, there is no knowledge of DSQL nodes and the processing done in the semantics phase. The engine does a parse of BLR, reconstruct another tree and compiles it. That does almost what DSQL does, but DSQL does it on SQL code and engine does it on BLR bytes.

When a statement is executed, the BLR tree is traversed and everything runs.

So back to DDL... DDL is not described as BLR, but in an equivalent binary format: DYN. Some DYN verbs have BLR bytes embedded, like the body of a stored procedure, but that is not relevant for generic DDL handling.

As in DML case, the engine knows nothing about DDL nodes constructed in DSQL. The engine knows only how to run DYN, with the isc_ddl API. In DSQL, everything works as in DML, but the generation phase emits DYN bytes, and statement execution is layered around isc_ddl.

So you might ask, what's wrong with DDL execution if it's so consistent with DML?

A lot of things, I'd say:

DYN codes are very badly structured. Some codes that would be private for some specific verbs shares the "number space" of all verbs. That make things visible wrong.

Related to above, we're going out of space. In v2.5, we're at number 247. We could go only at 254, and use the 255 to make a second number space. This would make things even worse.

While some may say that BLR is also bad, this is a model not invented in InterBase/Firebird. This is how compilers works (assembly, byte codes, etc). But that model does not have any practical benefit in DDL.

Too much code is used to convert DSQL structures to bytes and to engine again.

There is large code duplication to handle similar things, like CREATE / ALTER commands.

DYN execution is totally unnatural for a RDBMS system. A CREATE PROCEDURE command is more or less defined this way:

isc_dyn_def_procedure, <name>,
isc_dyn_prc_source, <source>,
...,
// for each parameter - begin
isc_dyn_def_parameter, <name>,
isc_dyn_prm_number, <number>,
isc_dyn_prm_type, <type>,
...,
isc_dyn_end,
// for each parameter - end
isc_dyn_prc_blr, <bytes>,
isc_dyn_end

And is executed in this way (pseudo-code):

Did you see what is wrong? Did you ever asked why Firebird system tables does not have primary and foreign keys?

The answer is simple, there is no way to have a FK from RDB$PROCEDURE_PARAMETERS to RDB$PROCEDURES if the records in RDB$PROCEDURE_PARAMETERS are inserted first.

With these problems defined, I started an alternate DDL path in v2.5, with the ALTER CHARACTER SET command. This command does not emit DYN, and when asked to execute, it does execute directly in DSQL with a new C++ friendly node (also used in the parser).

This was not sufficient through. To have benefits, existing commands shall also be migrated to the new scheme, and DYN should better be totally eliminated. The problem of this is that, documented or not, DYN and isc_ddl are part of the API, and used by client tools. The GDEF utility is totally based on it, as well DDL commands of GPRE.

These archaic utilities are the reason of how things had been defined in this way and why it never changed. Natively, former InterBase versions didn't understand text commands. DDL and DML had been part of client utilities, that compiles them and send BLR/DYN to the server.

Fortunately the Firebird team agreed to deprecate (in the sense of warn to tell to not use anymore) these things in v2.5, so it will be eliminated in v3.0.

In my branch for v3.0, I started to implement this conversion. Procedures and triggers had been refactored and their DYN handling was eliminated. The new external function was also done in the new scheme. With nice C++ classes, the architecture of PACKAGES became smooth. Also, the implementation of these commands became shorter, self-contained and much more readable.

Yesterday, I did a step further and did some unification of EXECUTE BLOCK (DML) with procedures and triggers (DDL) code. A prototype of sub-procedures is in the way...

CVSNT client is slow

2009-08-02T20:45:00.001-03:00

For years I was using CVSNT client to work with Firebird and any other CVS repository. I started to use it because I needed a client capable of handling the "sserver" protocol method, which the "standard" CVS can't.

A first thing I've noted has that it most times refetches completely (or just crashes trying) Firebird ChangeLog file during updates because some "corruption" (supposedly because it has more than 2MB). But that was OK with me, I just press Ctrl+Break and continue working.

By some reason, I had to use Cygwin's CVS to checkout whole firebird2 tree. It was much faster. So I switched to non-NT client and anything is much faster with it.

I've just started a merge of HEAD in my branch now. A lot of conflicts to resolve, most of them related to cleanups and formating changes. Unfortunately, it can't make this faster. :-)

BTW1, to open all conflicted files in gedit, I'm using:

grep -RlZ "<<<<<<<" builds/ src/ |xargs -0 gedit

BTW2, CVS is dead. Who cares?

BTW3, GIT seems excellent, but is somewhat difficult to grasp.

Influence of network latency in the Firebird wire protocol

2009-07-12T23:25:00.001-03:00

People often talk that Firebird performs badly in networks with high latency. I did some tests to see how the new protocol has been improved compared to the old one, and to see how it could be improved even more. My tests only cares about latency, and not about bandwidth.

It simulate network latency in Linux using Netem. With the tc utility, we can add a delay for outbound traffic. Since I did it for lo interface, it happens that it works for in/out traffic. I set delay to 100ms (that is more or less a value I found doing it in the internet with my horrible Telefonica ~~speed~~y connection), using this command:

$ sudo tc qdisc add dev lo root handle 1:0 netem delay 100ms

And here is the result of a ping:

$ ping localhost
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=200 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=200 ms
64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=200 ms
64 bytes from localhost (127.0.0.1): icmp_seq=4 ttl=64 time=200 ms

To later remove the delay, the command is:

$ sudo tc qdisc del dev lo root

The test uses Firebird library present on (my till unknown) CppSys project (more on CppSys in a future post). The CppSys database library has interfaces and semantics based on JDBC. The relevant test code is here:

println("Connecting to the database");
AutoPtr<FbDatabase> database(client->openDatabase(
   String("firebird:") + databaseName));

println("Starting the transaction");
AutoPtr<FbTransaction> transaction(database->startTransaction(
   TransactionIsolation::SNAPSHOT));

println("Preparing stmt1");
AutoPtr<FbPreparedStatement> stmt1(database->prepareStatement(
   transaction,
   "select rdb$relation_name from rdb$relations"
   "   where rdb$system_flag = 1"
   "   order by rdb$relation_name"));

println("Preparing stmt2");
AutoPtr<FbPreparedStatement> stmt2(database->prepareStatement(
   transaction,
   "select rdb$field_name from rdb$relation_fields"
   "   where rdb$relation_name = ?"
   "   order by rdb$field_name"));

println("Preparing stmt3");
AutoPtr<FbPreparedStatement> stmt3(database->prepareStatement(
   transaction,
   "select rdb$trigger_name from rdb$triggers"
   "   where rdb$relation_name = ?"
   "   order by rdb$trigger_name"));

println("Executing stmt1");
AutoPtr<FbResultSet> rs1(stmt1->executeQuery());

println("Starting fetch rs1");

while (rs1->fetch())
{
   println("rs1: %s", rs1->getString(1).getPtr());

   println("\tExecuting stmt2");
   stmt2->setString(1, rs1->getString(1));
   AutoPtr<FbResultSet> rs2(stmt2->executeQuery());

   println("\tExecuting stmt3");
   stmt3->setString(1, rs1->getString(1));
   AutoPtr<FbResultSet> rs3(stmt3->executeQuery());

   println("\tStarting fetch rs2");

   while (rs2->fetch())
       println("\trs2: %s", rs2->getString(1).getPtr());

   println("\tStarting fetch rs3");

   while (rs3->fetch())
       println("\trs3: %s", rs3->getString(1).getPtr());
}

The println function buffers the text and prints it in its next run, with the current time (in milliseconds) minus the time when the text was buffered. Here is its code:

static void println(const char* format, ...)
{
   static char lineBuffer[1024];
   static int64 lastTime = 0;

   timeb tb;
   ftime(&tb);

   int64 thisTime = int64(tb.time) * 1000 + tb.millitm;

   if (lastTime == 0)
       lastTime = thisTime;
   else
    {
       printf("%06d: %s\n", (int) (thisTime - lastTime),
           lineBuffer);
    }

   va_list va;
   va_start(va, format);

   vsprintf(lineBuffer, format, va);

   va_end(va);

   lastTime = thisTime;
}

As you see, the test does what every client/server database developer must knows that he/she shouldn't do: a main query with nested queries for each record. If you ever run ISQL over the internet, you had see it performs very slow, and that is the reason. ISQL does just that. It does that because it's written using embedded SQL with GPRE and sometimes this is needed to access new columns that may not be presented on a database of an older ODS. And due to embedded SQL nature, "details" of where and when to prepare statements are just ignored.

So far, here is the results.

Result for the old protocol, with 2.0 client and 2.5 server:

000665: Connecting to the database
000201: Starting the transaction
001234: Preparing stmt1
001204: Preparing stmt2
001204: Preparing stmt3
000200: Executing stmt1
000202: Starting fetch rs1
000000: rs1: MON$ATTACHMENTS
000200:    Executing stmt2
000201:    Executing stmt3
000201:    Starting fetch rs2
000000:    rs2: MON$ATTACHMENT_ID
000000:    rs2: MON$ATTACHMENT_NAME
000000:    rs2: MON$CHARACTER_SET_ID
000001:    rs2: MON$GARBAGE_COLLECTION
000000:    rs2: MON$REMOTE_ADDRESS
000000:    rs2: MON$REMOTE_PID
000000:    rs2: MON$REMOTE_PROCESS
000001:    rs2: MON$REMOTE_PROTOCOL
000000:    rs2: MON$ROLE
000000:    rs2: MON$SERVER_PID
000000:    rs2: MON$STATE
000001:    rs2: MON$STAT_ID
000000:    rs2: MON$TIMESTAMP
000000:    rs2: MON$USER
000601:    Starting fetch rs3
000000: rs1: MON$CALL_STACK
000200:    Executing stmt2
000201:    Executing stmt3
000201:    Starting fetch rs2
000000:    rs2: MON$CALLER_ID
000000:    rs2: MON$CALL_ID
000000:    rs2: MON$OBJECT_NAME
000000:    rs2: MON$OBJECT_TYPE
000000:    rs2: MON$SOURCE_COLUMN
000000:    rs2: MON$SOURCE_LINE
000000:    rs2: MON$STATEMENT_ID
000001:    rs2: MON$STAT_ID
000000:    rs2: MON$TIMESTAMP
000601:    Starting fetch rs3
<...>
000000: rs1: RDB$CHECK_CONSTRAINTS
000200:    Executing stmt2
000200:    Executing stmt3
000200:    Starting fetch rs2
000001:    rs2: RDB$CONSTRAINT_NAME
000000:    rs2: RDB$TRIGGER_NAME
000200:    Starting fetch rs3
000000:    rs3: RDB$TRIGGER_14
000000:    rs3: RDB$TRIGGER_15
000001:    rs3: RDB$TRIGGER_16
000400:    rs3: RDB$TRIGGER_35
000000: rs1: RDB$COLLATIONS
<...>

Result for the new protocol, with 2.5 client and 2.5 server:

000670: Connecting to the database
000200: Starting the transaction
000229: Preparing stmt1
000203: Preparing stmt2
000203: Preparing stmt3
000000: Executing stmt1
000202: Starting fetch rs1
000000: rs1: MON$ATTACHMENTS
000000:    Executing stmt2
000000:    Executing stmt3
000202:    Starting fetch rs2
000000:    rs2: MON$ATTACHMENT_ID
000000:    rs2: MON$ATTACHMENT_NAME
000000:    rs2: MON$CHARACTER_SET_ID
000000:    rs2: MON$GARBAGE_COLLECTION
000000:    rs2: MON$REMOTE_ADDRESS
000000:    rs2: MON$REMOTE_PID
000000:    rs2: MON$REMOTE_PROCESS
000000:    rs2: MON$REMOTE_PROTOCOL
000000:    rs2: MON$ROLE
000000:    rs2: MON$SERVER_PID
000000:    rs2: MON$STATE
000000:    rs2: MON$STAT_ID
000000:    rs2: MON$TIMESTAMP
000000:    rs2: MON$USER
000200:    Starting fetch rs3
000000: rs1: MON$CALL_STACK
000001:    Executing stmt2
000000:    Executing stmt3
000200:    Starting fetch rs2
000000:    rs2: MON$CALLER_ID
000000:    rs2: MON$CALL_ID
000000:    rs2: MON$OBJECT_NAME
000001:    rs2: MON$OBJECT_TYPE
000000:    rs2: MON$SOURCE_COLUMN
000000:    rs2: MON$SOURCE_LINE
000000:    rs2: MON$STATEMENT_ID
000000:    rs2: MON$STAT_ID
000000:    rs2: MON$TIMESTAMP
000200:    Starting fetch rs3
<...>
000000: rs1: RDB$CHECK_CONSTRAINTS
000000:    Executing stmt2
000000:    Executing stmt3
000201:    Starting fetch rs2
000000:    rs2: RDB$CONSTRAINT_NAME
000000:    rs2: RDB$TRIGGER_NAME
000200:    Starting fetch rs3
000000:    rs3: RDB$TRIGGER_14
000000:    rs3: RDB$TRIGGER_15
000000:    rs3: RDB$TRIGGER_16
000000:    rs3: RDB$TRIGGER_35
000000: rs1: RDB$COLLATIONS
<...>

FWIW, I did tested the old protocol on real network with similar latency too. The result was very resemblant.

The total times are:

Old protocol	55.5s
New Protocol	18.5s
New Protocol minus round-trip times of rs3	10.1s

So we may see that:
1) The old protocol is a crap.
2) The new protocol does a good job removing some "unnecessary" round-trips.
3) It may be improved.

The first interesting problem is the series of statement preparations. The preparation of a statement may result in an error, so it necessary involves a round-trip so we can get the status from the server. But does it really needs it? Using Oracle JDBC driver, a prepareStatement with a wrong statement does not throw an exception. It's deferred. In fact, this is a contract, and a change in a contract would require some way for the client applications to enable or disable it.

The second interesting thing is that executing a statement that needs fetch does not involves a round-trip. Does it happens too when selecting from a SP? Won't this be a contract change as well, since selectable SPs could write to the database? I didn't did further tests... Anyway, that's what allows the optimizations I talk below.

Then we start the fetches. The first fetch for rs1 necessarily involves a round-trip and buffers some records in the client. Subsequents fetches are resolved locally.

The third interesting thing is about the nested queries. Since queries are executed, a round-trip would be inevitable, but it does one for each query. A more intelligent approach would be to fetch rs3 too when asked to fetch rs2, since it's already executed. In this test, it would run ~8s faster. And a much more intelligent approach would do it in the background (and transmitting asynchronous), without putting some overhead on the rs2 call.

With a super intelligent server/client/protocol, the client and the server may cache prepared statements, and the server may asynchronous send statements invalidations to the client. Then, preparation of statements would have no round-trip (when already prepared one time), being sure that the statement is not wrong.

So what you may do now?
1) Do not talk with the server in this poor (and common) way. Use selectable SPs to run multiple queries.
2) Do not prepare/close/prepare your common statements. Cache them in your application.