Avisynth+

[ultim]

On a (not-so-) unrelated note, given that Avisynth's development has been mostly dormant with only very few contributors, it is not very motivating when somebody steps up with work and willingness to make the situation better, to tell him "go and work on that other project". Especially because nobody has said anything wrong about my proposals. If it had problems like breaking existing functionality or making code worse I'd understand. Or if IanB had reviewed them and told me "you should fix that differently because...", or any kind of constructive criticism, I'd go back and fix those things ASAP. But literally the only problem some of you had with my changes is that "there are too many", which makes me LOL in sadness. Should I beg for forgiveness that I contribute to the project at all?

[MasterNobody]

ultim
I think problem with review is not that you have too many functional changes but due your move/rename files commits. You can't really review/apply patches after it if you not going to apply this file structure refactoring patch (and as I understand IanB don't want to change file structure only because you don't like it). And so you can't cherry pick only ok commits. This way patches have "all or nothing" condition to apply which is not good.

[ultim]

There are multiple solutions to this problem, off the top my head (without any particular order of preference):

• IanB could check out my repository in Git without merging any patches, to look at the directory structure I have arrived at. If he doesn't like the placement of some files, he can make suggestions to me how I should rebase the code, and ofc I'd be willing to help.
• IanB could at least look at the directory structure I have arrived at. Maybe he'll come to the same conclusion that the new structure really is better than the old one, in which case he can merge the code. Selectively picking commits after the commits that move files (which come at the very beginning) is not a problem any more in this case.
• IanB could reorder the files as I did, then selectively pick commits (which will be trivial since he moved the files too), and then move the files back to their original places. He'll need to fix up some #include lines, but this is basically not much work either. But this method allows him to cherry-pick my commits without accepting the new directory structure.

So the only thing left to 'solve' is the manual work needed to actually move the files for the first commit. This can be done fully automatically, even if he is using CVS, because all my work is based on Ian's latest commit. All he needs to do is copy his CVS repo into a new folder, delete the sources without deleting the history, and then check out my Git repo into the same folder. Bang, no need for patches, no need to merge, and no need to move files manually. At this point he can start committing to his CVS if he decides he likes my work. And if he does not like it (after inspecting it), he still has his own repository untouched.

The problem is, is that IanB does not even want to take a look if I have created a logical directory structure. He simply decided that he will not take a look at my changes just because I have moved files, even though they do not hide my changes, because I kept moves from changes separate.

Note that looking at the first commit of mine looks much scarier than it really is. A lot less files have been moved than it looks like, because a large part of the additions come from not moves, but from adding the HTML sources of the FilterSDK to the repository. 99% rest of the moves are whole directory moves, like the 'src' folder being renamed to 'core', or moving the pfc and soundtouch directories into plugin folders. I think you can probably count the number of files I have moved within the core itself on one hand. The rest are as described.

Please, tell me honestly: If you compare the structure of the CVS repository and my Git, isn't the new folder structure much better? Because if it is, then this whole discussion is meaningless. And if it is not better, I have still described two methods in the bulletpoints above, how Ian can fully or partially reject my folder structure while still picking other commits from me.

[Guest]

Here's another option. Make your patches to the existing directory structure.

[ultim]

Originally Posted by neuron2

Here's another option. Make your patches to the existing directory structure.

That falls within my first bulletpoint. And honestly, I'd even be willing to comply (even if not completely happy) if he said, that my directory structure was looked at seriously, but he found it inferior to the original because such and such. I don't think this is an unreasonable request from a contributor, to at least take a look at his work, and only reject it if there are founded objections. This goes for moving files too.

What I am disappointed at, is if Ian decided not to inspect my new structure just because, out of spite. Unfortunately, from his last post it looks like that this is pretty much the case, but I still have some hope left. I hope he'll post soon again to let us know his course of action.

[Guest]

Saying IanB rejects your stuff out of spite is way over the top and won't endear you to our members!

Maybe he doesn't even have GIT, or doesn't have the time to come up to speed on it. Personally, I would have to spend (waste?) many hours just to get to the point of being able to look at your changes.

And you are asking *him* to undo your directory structure changes?! There's some major chutzpah there, no?

[ultim]

Originally Posted by neuron2

Saying IanB rejects your stuff out of spite is way over the top and won't endear you to our members!

Maybe he doesn't even have GIT, or doesn't have the time to come up to speed on it. Personally, I would have to spend (waste?) many hours just to get to the point of being able to look at your changes.

And you are asking *him* to undo your directory structure changes?! There's some major chutzpah there, no?

Please don't stir up emotions. If you read my previous post carefully, you'll see that I have written, "... if Ian decided to...", which is hypothetical, and I ended my post by saying I hope this is not so, and that I hope for the better. I never said that he is doing it out of spite, I said I am hoping he is not.

I also wasn't asking him to undo my changes, it was one of the multiple possibilities, just like the one where he tells me what to change, and I'd do it myself (1st point). Grabbing isolated sentences out of my post and changing the overall meaning is not fair. Again, I'm trying to cooperate here, offering further work to fix what you ask me to, all I ask for is a fair review with arguments. There is no need to install Git or spend hours just to start taking a look as you mention it, because you can browse the source online on GitHub.

So no, no chutzpah.

[Guest]

You said this:

"if Ian decided not to inspect my new structure just because, out of spite. Unfortunately, from his last post it looks like that this is pretty much the case"

That is crystal clear to anyone with a brain.

[ultim]

I also said that "grabbing isolated sentences out of my post and changing the overall meaning is not fair", and you left off the end of the other sentence, which is again, changing the meaning. But since you look like you honestly misunderstood it, I'll say: Sorry. I'm, sorry. I didn't mean it that way, but because you obviously don't believe me, let me apologize.

[ultim]

Please let me try to express myself just once more, one last time, because there was some bad choice of words in the discussion with neuron2 that lead to misunderstandings.

I have made multiple commits over the course of a month, IMHO improving Avisynth's code in many different aspects. When submitting my changes, I ...

• published my changes with full history on GitHub,
• I based my work on the latest commits by upstream,
• I have clearly separated commits,
• I moved files in different commits than I made changes, so that changes are not obscured,
• I am willing to further patch up my work based on suggestions after a review,
• and the code I submit is commented.

Hence, I think that my commits are upstream-friendly, with the worst offender being the file moves. The friction between us is that you tend to view my changes as a bunch of patches, where I, unfortunately, messed up by reorganizing files before the patches. However, I'd like you to look at my file moves as part of the improvements too. I didn't reorganize the folder just because I was feeling like it. They are meant to be part of the improvements, and I feel the new folders make the project structure much more logical, because:

• I have moved files and libraries which are part of the core into the core.
• I have moved files and libraries which are NOT part of the core out of the core.
• I have separated non-essential filters into plugin DLLs.
• I have reduced the dependencies on 3rd-party code.
• And, by all the above, I also have made it easier for new contributors to find the files they are looking for,
• and easier to get a grip on the project.

So, me moving files around is meant to be part of the changes, not some extra stuff that I messed up by chance. And I sure have not "randomized" the locations like IanB put it.

This work took me a month, and as an example, my whole last weekend. In total more than a full day was spent merging, rebasing and cleaning up commits, because there were orignally about 1.5x-2x as many. So yes, I think I've been a good and diligent contributor. The only "bad" things left are the file moves, but I'd prefer not to get rid of them as they ARE part of the improvements, because of the numerous reasons that I have listed above. (And yes, please also note that I can argument why the folder structure is better this way.)

And when I publish my work, people tell me they don't even want to take a peek at any of my work just because they wouldn't have moved files in the first 2-3 commits (which are also improvements), even though it would take them only a fraction of the time that I needed to implement the whole thing. Whereas, if they did, they might even find that the moves make sense.

I hope I managed to bring you closer to my point of view.

[Keiyakusha]

I haven't read the whole stuff, only last post and I don't quite get it. You say "published my changes with full history on GitHub". Well ok, and what do you expect to see after that? If I would be on IanB's place I wouldn't even move my ass unless you'll send me patch for my own repo that I can review and apply. Maybe if avisynth was a super-active project with some devs that have tons of motivation... But that's not the case.
Whatever you fork it and maintain your own repo, or provide patches that can be easily applied. Dunno about IanB, but personally I don't even know how git works. I usually use SVN for my stuff which does everything that I need, so I don't want to learn git, nor clutter up my system with useless software. If things are the same for IanB (just with CVS in this case) I can understand this.

[IanB]

@Ultim,

Okay I made some time to look at your repository. From that I found 3 new very minor bugs. Very poor time invested per bug. I don't understand why you did not just reported them back in post #189 on 1st September 2013 instead of complaining about my going all red on you. Sorry you took the red text personally it was aimed at a much wider audience, it's a sad fact some simply cannot be bothered succinctly reporting the bug they find. Fortunately there are also many who do actually make the effort.

Yes I can see you have many ideas. Some I had already thought about for future versions, some new ones that I will take on board. Congratulations you are the first other person to recognise that we need _set_se_translator for MSVC versions later than 2K3.

It's a genuine pity that you could not resist the urge to move files around. You have a certain biased perspective (i.e it's Ultim's way only) that we did not share in so your restructuring just creates loss of reference without any personal benefit.

[ultim]

Originally Posted by IanB

It's a genuine pity that you could not resist the urge to move files around. You have a certain biased perspective (i.e it's Ultim's way only) that we did not share in so your restructuring just creates loss of reference without any personal benefit.

*Sigh* You still don't give any reason why the files should not be moved, or why the places I moved them to are the wrong ones.

And please don't tell me that I only accept my way. I see it quite the contrary. I have written down literally half a dozen arguments why the structure matters and why the new structure is better, and even told you that I'll use any other structure as long as there is a reason for it. On the other hand, you just tell me "No, I won't move any files" without giving any justification. And of the two of us, I'm the biased?

But ok, I guess this is the most I will get. So I'll try to approach it from a different perspective. You told us yourself a couple of posts back that you've been itching to move file too, but you resist. Why do you resist? It is not like there are other code bases depending on your CVS, except for AvisynthMT.

So maybe after consulting with SEt, we could do it according to your dream-structure. Because it sure doesn't have to be my structure, I'd just like it to change, because the current structure sure is bad right now.

Maybe some background why I find it important to change: My goal is not just for me to improve Avisynth, but to allow it for more people from the community to join the development too. Currently it is very hard for anyone to do so, because of multiple reasons, like: unorganized files, the outdated development environment (VS6), unreadable code, missing commenting, the huge amount of exceptions thrown, and the choice of CVS is not the most contributor-friendly either.

This is why my current changeset focuses on improving developer friendliness, so that we lower the barrier for entry for new contributors. I understand, that unfortunately this means changes for existing developers too, mostly for you. But if we manage to attract more developers over time, this will pay off. The project structure plays a large role here. When a newbie comes, and sees source .cpp file of the core being in the distrib folder, some parts of non-essential plugins in the core, others parts in completely unrelated directories, "include" directories containing source files etc., he'll be confused, and with every right!

So, IanB, can we maybe reorganize the files according to how you've been wanting to have them?

[qyot27]

Some basic things related to the git history that I find I do want to comment on now as actual feedback, related to the file structure and general practices with git (and this also applies to generating out patches with 'git format-patch' for submission upstream).

I do feel the new structure is an improvement, but it's the presentation that I think needs to be done a bit better so it can be discussed and a more solid case can be made for why the individual choices made were made.

The renames, moves, adds, and deletes should be split into many smaller and separate commits, and the commit messages should fully (or as fully as possible) explain what is happening in each one, such as where files are getting moved/renamed to, or why binary files are deleted, etc.

Things that should have their own dedicated commits, based on what I've observed about the new structure:

• Expanding the Examples out of their .zip archive
• Deleting binary files from the git repository/history
• The basic rename of the 'src' directory to 'avs_core' (and nothing else); this is probably 90% of the difference in the first place
• Adding the docs to the history and noting where they are stored
• Adding the CMake build system (if the CMake files for the newly-split plugins is purely for making these into separate plugins, leave those CMake files out for now; if they actually do have relevance for basic compilation as traditionally done, leave them in)
• Removing the old build system's files, and explaining why they are being removed
• Minor file renames - and make sure the renames are listed in the commit message itself.
• Reorganizing of files to more appropriate locations (such as putting all source files related to converting things in the 'convert' directory); this should be separated into as many distinct commits as there are distinct subtopics for reorganizing (in other words, in the convert example I just gave, that is one commit; if there's another one related to some other destination directory, do it in a second commit, and so on)
• Making Shibatch/ImageSeq/SoundTouch their own plugins (and each one should have its own commit) - IMO it's okay if any renaming and moving is consolidated into here in one patch, just so long as the moving/renaming is immediately pertinent to the goal of making the plugins distinct, and the rationale is - once again - explained in the commit message

Having these split into distinct commits makes it easier to take it piece by piece, explain the rationale for each change, and get approval or critiques from the upstream devs on what should be done. Explaining things in the commit messages and keeping them small in subject matter also doesn't give off the impression of a dramatic change and it's easier to see the immediate connection between where a file was and where it was moved to.

Another point more immediately related to git itself, is that the formatting of the commit messages should have a single concise summary as its headline, and then a longer explanation further down after a double-space. The lines in the commit message should be wrapped at 80 characters (the headline should be <= 80 characters in length). This is to make it easier for users pulling up the history with git log to view it without it overflowing their Terminal's width buffer (since most default to 80 characters), or for any need to use git rebase -i - if the lines exceed the width buffer, then it has the potential to interfere with git rebase -i in a really bad way.

Also, somewhat of a minor point, but let's say you want to focus on the development of a single issue, like multithreading. The work on this should be in its own branch, separate from master. This keeps the master branch clean so that integration of newer changes from upstream and rebasing the branches against master is easier and keeps the general noise down (IMO, development branches should always be considered volatile and subject to rebasing, even if the git repository is public; but never, ever, ever rebase master).

So another list of general git management points I think would make this easier:

• Checkout a new branch for the changes you've made. All 54 of them that are currently on the github repo should be in a separate branch, and then reset --hard master's HEAD back to where upstream leaves off
• Use git rebase -i on the new branch to edit the commit messages to comply with the 80 char limit and separate headline/explanation format, without changing the contents of any of the actual commits.
• Use git rebase -i again, but this time to split the file structure changes into many smaller, more topical commits.

Also, I think there should be a new thread to discuss the changes ultim is proposing, so that the alpha4 thread doesn't get dragged any more off-topic (especially now that alpha5's been released).

[ultim]

Currently, the reason everything is on the main branch in my repo, is because it is my repo. Meaning it is set up so that if anyone clones my master, he'll get the latest stable version of my changes. That makes the most sense to me. Why would anyone clone my repo if they want the official branch of Avisynth? So my stable work is in the master of my repo, while I'd be using branches for non-stable or work-in-progress of other features I'm working on, and for upstream. This is not a problem for IanB, because he can pull changesets/patches from any of the branches, his master does not need to be the same as my master branch. If I were in the official repo though, I'd obviously be working in a seperate/private branch, though I doubt IanB will let me do that.

Concerning your other idea of splitting up the large file-moving commit, I'm okay with that. As long as I know beforehand that IanB is not gonna reject it again, I'd redo my changeset and split it up further as you proposed. So if he lets me know that he's willing and ready to integrate/merge the file moves that way, count me in. Of course we can discuss where files should go before I start redoing everything, so that I won't have to do it a 3rd time.

[qyot27]

Originally Posted by ultim

Currently, the reason everything is on the main branch in my repo, is because it is my repo. Meaning it is set up so that if anyone clones my master, he'll get the latest stable version of my changes. That makes the most sense to me. Why would anyone clone my repo if they want the official branch of Avisynth? So my stable work is in the master of my repo, while I'd be using branches for non-stable or work-in-progress of other features I'm working on, and for upstream. This is not a problem for IanB, because he can pull changesets/patches from any of the branches, his master does not need to be the same as my master branch. If I were in the official repo though, I'd obviously be working in a seperate/private branch, though I doubt IanB will let me do that.

That suggestion was mostly to make sure that there's no potential conflicts if you try to pull in changes from upstream - having all of it in master means that there's the potential for changes to be sandwiched between runs of cvsimport, or for conflicts in the tree to arise that might be more unexpected (for that matter, the stuff from cvsimport could go in a different branch like 'trunk' or something, with master the way it currently is; the same basic idea applies - letting whatever happens upstream have its own branch). Github does allow the ability to set alternate branches as main so that a user doing a clone gets the branch you want them to get (I had it set up that way for my FFMS2* repo for a long time, where the 'patches' branch was the one you'd get from a normal clone).

*which is now the ffms2-old repo; I re-forked my main one against the official repo after the move to Github

[ultim]

Very unfortunately, me and IanB seem to have reached an impasse. I have received constructive PMs though from forum readers, some relating to how I present my patches, some regarding communications, but all encouraging. I'll be continuing my slow-ish efforts at improving Avisynth, and because upstream acceptance does not seem realistic ATM, I'll be forking the project (well, technically speaking, I already have). I wanted to avoid this, because forking has many downsides. Downsides like even more fragmentation, a lessened likelyhood of both projects benefitting from each other's improvements, and a bitter aftertaste in the mouth on both sides, I guess. The more optimistic amongst you can however spot advantages too, like not having to be constrained by the ideals and rules of the father project. I'll post more information about goals and such in my own thread.

Anyway, I just wanted to give you an update, to prevent you from thinking that I'm gone. This will be my last post "contaminating" this thread, and I'll create a thread of my own the next time, which will be when I post my next release. I have already synched with 2.6 alpha5 (synching was not a big deal), and have a large chunk a new code in place which completely (like, from scratch) replaces the plugin handling and loading. Also, Groucho2004 was right after all when he found something was off with out-of-memory handling in my build, so we'll see what I can do about that. It has to do with me using set_seh_translator and /EHa, which IS the correct thing to do on all VS2003 and above, but AFAICT it causes unhandled std::bad_alloc exceptions not to be swallowed silently anymore... which the project seemingly has relied upon? I've still got to take an exact look at that, but I know I'm on the right track. Anyway, back to my plugin-handling code, it makes script loading percievably faster if you have a lot of plugins, and it is also much cleaner from a developer's perspective. Oh, and expect support for multiple plugin directories too.

To finish forking, I guess the only thing left to do is to find a name for the fork. Feel free to send me any suggestions, I am torn between using something "Avisynth"-like or not. First of all, it'd be natural to do it, because my fork is Avisynth in its core. Admittedly, I am still relying on a large existing code base, so it'd be misleading to try to present the fork as something completely different. On the other hand, it would also be aggressive and disrespectful from me to have a name like Avisynth 3-4-5-6..., Avisynth-X, -NG, or who knows what suffix. So let me know what you think or if you have any ideas. I'm very interested in Ian's oppinion too, so I'd be thankful if he shared his preferences on naming with us. Of course I'm not asking him to come up with specific names for me, I'd just like to know what kinds of names are okay for me to use from an official perspective.

Cheers

[Groucho2004]

Originally Posted by ultim

Also, Groucho2004 was right after all when he found something was off with out-of-memory handling in my build, so we'll see what I can do about that. It has to do with me using set_seh_translator and /EHa, which IS the correct thing to do on all VS2003 and above, but AFAICT it causes unhandled std::bad_alloc exceptions not to be swallowed silently anymore... which the project seemingly has relied upon?

Interestingly, SEt's MT DLL (built with VC2010) exhibits the same exception handling behaviour as the official one that is built with VC6.
No idea what he did, but I think he mentioned something about "hacking" the compiler.

[ultim]

Originally Posted by Groucho2004

Interestingly, SEt's MT DLL (built with VC2010) exhibits the same exception handling behaviour as the official one that is built with VC6.
No idea what he did, but I think he mentioned something about "hacking" the compiler.

That's because he is using different compiler settings. If I switch to /EHsc (which is the theoretical equivalent of VC6's /GX), I get similar exception behavior as in the official build. Of course then set_seh_translator falls apart. The official project solved set_seh_translator's absence using custom ASM-hacks, and I guess those are the hacks that SEt referred to. I got rid of those hacks in my build, as I prefer to use methods that are officially supported by the compiler. But now I need to fix up parts of the code to properly support this scenario.

On the other hand, it would also make sense to completely remove handling SEH-exceptions, because there are very few valid reasons to handle them, and even less that are worthy of showing to the user, as they indicate coding errors in Avisynth or in the plugins. I might really remove them in a later build of mine, but not now, to take it step-by-step.

[Groucho2004]

Originally Posted by ultim

That's because he is using different compiler settings. If I switch to /EHsc (which is the theoretical equivalent of VC6's /GX), I get similar exception behavior as in the official build.

Not true for Windows XP. I tried that with VC7.1, VC9 and VC10, I always get this silly runtime message box instead of the actual message from "ThrowError()". I think it has something to do with msvcrt.dll on XP, not sure.

It would be nice if Avisynth could be built with newer compilers since there certainly have been improvements since VC6 but on the other hand, I have not come across any problems for which VC6 could be blamed exclusively.